Why an Ordering Is Needed
In a group sequential test, the reported result includes more than a test statistic: it also records when the trial stopped. A statistic of 2.1 at the first look and a statistic of 2.1 at the final look are distinct stopped outcomes, even though their numerical values match. To define a p-value, we need a rule for deciding which possible stopped outcomes count as at least as extreme as the observed one.
The ordinary fixed-sample rule compares a statistic with its null tail. After stopping, that rule can miss probability from other looks and may not correctly describe the chance of obtaining a result ranked as strong as the observed result. The earlier material on stopped outcomes and superuniformity provides the key setup: the stopping look and the statistic together determine an outcome, and a valid p-value must be calibrated under the null distribution of that stopped outcome.
An ordering can be expressed by an evidence score \(R(j,z)\), where larger values mean stronger evidence. Then the at-least-as-extreme set is \(\{(k,x):R(k,x)\ge R(j,z)\}\). The rule must be defined across looks: it is not enough to say which values of \(Z_j\) are extreme at the observed look while ignoring outcomes at other looks.
From an Ordering to a Null Tail Probability
Once the ordering is fixed, the p-value is the null probability of the full at-least-as-extreme set. In a discrete outcome space, this probability is a sum. If outcomes have continuous statistics, it is an integral over the stopping regions and statistic values satisfying the ordering rule. In either case, probability is computed under the null model for the actual stopping procedure, not under a hypothetical procedure that always continues to the observed look.
The notation \(P_{H_0}\) means probability under the specified null model, including the rule that determines \(J\). For a continuous statistic and look \(k\), the probability contribution comes only from paths that actually stop at \(k\). For example, a path that would have crossed an earlier efficacy boundary cannot contribute to the probability of a final-look outcome.
This definition alone does not select one unique ordering. A rule must say how evidence at different looks is compared. Different coherent rules can put different outcomes in the tail and therefore produce different p-values for the same observed pair. The ordering should be justified for the inferential question, specified without choosing it to make the observed result look more significant, and calibrated using the null distribution of the stopped procedure.
Worked Example: Summing the Tail Under a Specified Ordering
Consider a hypothetical design whose stopped outcomes are grouped into six categories. The entries below are null probabilities for the categories; they sum to 1. For this illustration, the prespecified evidence score is \(R\), with larger scores indicating stronger evidence.
| Stopped outcome category | Null probability | Evidence score \(R\) |
|---|---|---|
| Look 1, lower statistic | 0.50 | 0 |
| Look 1, moderate statistic | 0.20 | 1 |
| Look 1, high statistic | 0.05 | 2 |
| Look 2, lower statistic | 0.15 | 0 |
| Look 2, moderate statistic | 0.07 | 1 |
| Look 2, high statistic | 0.03 | 2 |
Suppose the observed outcome is Look 2, high statistic, with score \(R=2\). At least as extreme means \(R\ge2\), so both high-statistic categories enter the tail, not just the category at Look 2. Thus
Under this specified ordering and null distribution, the p-value is 0.08. The result is not significant at the 0.05 level. The calculation uses probability at both looks because both categories meet the ordering criterion.
Why the Null Stopping Distribution Matters
The distribution of \((J,Z_J)\) is generally not the same as the distribution of a statistic observed at a fixed look. In a group sequential design, reaching a later look means the path has satisfied the earlier continuation rules. Consequently, the null probability of a final-look result must account for that survival condition. This is why the p-value tail is a probability over stopped outcomes, rather than simply a tail probability for \(Z_j\) considered as if look \(j\) had been fixed in advance.
For a continuous statistic, let \(f_k(x)\) denote the null density for observing \(Z_k=x\) and stopping at look \(k\), including the probability of reaching that look. If the ordering score is \(R\), the p-value can be written as
Here \(K\) is the maximum number of looks. The density \(f_k\) is a stopped-outcome density, not merely the marginal density of \(Z_k\). This expression makes the two ingredients visible: the ordering determines the integration region, while the null stopping distribution supplies the probability assigned to that region.
Worked Example: The Ordering Can Change the Tail
Use the same hypothetical null probabilities from the previous table. Suppose the observed result is Look 1, moderate statistic, with probability 0.20. Under the score ordering in that table, its score is 1. The at-least-as-extreme categories have scores 1 or 2, so
Now consider a different, explicitly hypothetical ordering that ranks categories only by their statistic band: lower, moderate, or high, with no adjustment for look. The observed moderate band is then at least as extreme as all moderate and high categories. The tail probability is
In this particular table the two orderings happen to give the same tail probability for this outcome. That equality is not guaranteed. For the observed Look 2, high-statistic category, the first ordering gives \(0.05+0.03=0.08\). A different rule could place only that specific category in the upper tail, giving \(0.03\), but such a rule would need a substantive justification and must be specified independently of the observed result. A smaller tail created by an ad hoc ordering is not automatically a more meaningful measure of evidence.
Ties, Validity, and a Short Justification
When multiple outcomes have the same evidence score, the usual nonrandomized definition includes all of them. Thus the comparison is \(R\ge R(j,z)\), not just \(R>R(j,z)\). Omitting tied outcomes generally reduces the tail probability and can destroy the superuniformity required of a valid p-value. In a continuous setting ties may have probability zero under some score rules, but discrete statistics and rounded or grouped outcomes can produce ties with positive probability.
The basic validity justification is an ordering argument. Under \(H_0\), list the distinct evidence-score levels from highest to lowest. The p-value at a score level is the cumulative null probability of that level and all higher levels. The event that the p-value is at most a chosen value \(a\) can include only score levels whose cumulative probability is at most \(a\). The total probability of those levels cannot exceed \(a\). Therefore, for every \(0\le a\le1\),
This is superuniformity. It allows conservatism, which commonly arises when score levels have positive probability and the tail probability jumps rather than varying continuously. It does not imply that every possible ordering is scientifically appropriate; it says that a fixed ordering paired with the correct null distribution gives a calibrated tail probability.
Worked Example: Inclusive Tails with a Tied Score
Suppose a hypothetical stopped-outcome distribution has three evidence-score levels. The null probabilities at scores 2, 1, and 0 are respectively 0.06, 0.14, and 0.80. An observed outcome has score 1. The inclusive tail contains every outcome with score at least 1:
All outcomes tied at score 1 are included, regardless of their stopping look. If one instead used the strict tail \(R>1\), the reported value would be 0.06. But under the null, every outcome with score 1 would then receive that same strict-tail value, so
That violates superuniformity at \(a=0.06\). The inclusive value 0.20 avoids this problem by counting the entire tied score level.
Practical Checks and Limits
Before interpreting an ordering-based p-value, check that the stopping rule and null model used for its calculation match the design. Verify that probabilities over all possible stopped outcomes sum to 1, and check that each look’s contribution includes only paths that reach and stop there. Numerical integration or recursive calculations can be checked against total probability and, where feasible, simulation under \(H_0\).
A p-value depends on both the ordering and the null distribution. It is not the probability that \(H_0\) is true, and it does not state the probability of observing exactly the reported outcome. It is the null probability of the set defined by the chosen ordering. A different ordering can be defensible for a different inferential purpose, but the resulting p-values need not agree.
A common mistake is to compare the observed statistic only with the null tail at its stopping look, as if that look had been fixed in advance. The correction is to define the evidence ordering across the full stopped-outcome space, then sum or integrate the null probability of every outcome that ranks at least as strongly as the observation. A second mistake is to choose the ordering after seeing which one yields the smallest p-value. That makes the reported tail depend on the data in an unaccounted-for way; the ordering must be fixed or its selection must be incorporated into the calibration.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Why does an observed outcome need to include the stopping look as well as the statistic?
- Under a fixed evidence score, what does the at-least-as-extreme set contain?
- A discrete score assigns null probability 0.04 to score 3 and 0.11 to score 2, with higher scores stronger evidence. What is the inclusive p-value for an observed score of 2?
- Why can a final-look statistic’s ordinary fixed-look tail probability fail to give the stopped-outcome p-value?
- What property should the null distribution of a valid p-value satisfy?
Solutions. (1) The stopping look is part of the realized outcome, and the null probabilities can differ across looks. (2) It contains every possible stopped outcome whose score is at least the observed score. (3) \(0.04+0.11=0.15\), including all ties at score 2. (4) The fixed-look calculation can ignore earlier stopping and the probability of reaching the final look; the stopped-outcome tail must include the actual stopping rule. (5) It should be superuniform: \(P_{H_0}(p\le a)\le a\) for every \(a\) between 0 and 1.