Why compare two orderings?
After a group sequential trial stops, the observed result includes both the data and the look at which monitoring ended. A p-value must therefore assess the observed stopped outcome against the null distribution that accounts for the stopping rule. There can be more than one defensible way to decide which other stopped outcomes count as at least as extreme as the observed one.
A stagewise ordering ranks stopped outcomes according to a prespecified stagewise evidence measure. A likelihood-ratio ordering ranks them according to how much more likely they are under a specified alternative than under the null. These orderings answer related, but not identical, questions. They can assign different ranks to the same outcomes and consequently yield different p-values.
The meaning of “at least as extreme” depends on the ordering. For a stagewise p-value, the ordering is specified by the stagewise evidence rule. For a likelihood-ratio p-value, outcomes with larger likelihood ratio count as at least as extreme. The null distribution used to calculate either tail must include the actual stopping rule and look schedule.
Define the two orderings before calculating
Write \(S_{\mathrm{SW}}(y)\) for a stagewise evidence score, with larger values defined to mean stronger evidence against \(H_0\). The score can depend on the stopping look and the observed data. The corresponding stagewise p-value is \(P_0\{S_{\mathrm{SW}}(Y)\ge S_{\mathrm{SW}}(y)\}\). This probability is over all possible stopped outcomes, not merely outcomes from the look at which the observed trial stopped.
For a likelihood-ratio ordering, specify an alternative distribution \(P_1\), often corresponding to a particular effect value under \(H_a\). For outcomes with positive null probability, define \(L(y)=P_1(y)/P_0(y)\), or the ratio of the alternative and null densities in a continuous model. Larger values of \(L(y)\) favor the specified alternative relative to \(H_0\). Its ordered p-value is \(P_0\{L(Y)\ge L(y)\}\).
For a simple alternative and a stopped outcome \(y\), the likelihood ratio uses the probability or density of the entire observed stopped data under each hypothesis. If the stopping rule is fixed in advance and does not depend on the hypothesis, the restriction that the observed path reached its stopping look is part of the event space under both hypotheses. It must not be dropped when forming the null tail probabilities.
Worked Example: Different Rankings, Different P-Values
Consider a hypothetical sequential experiment with five mutually exclusive, exhaustive stopped-outcome categories. The probabilities below are joint probabilities after applying the stopping rule. A prespecified stagewise score gives the categories the displayed stagewise order. For illustration, take \(P_1\) to be one specified alternative.
| Outcome | Stopping look | Stagewise rank | \(P_0\) | \(P_1\) | \(L=P_1/P_0\) |
|---|---|---|---|---|---|
| A | 1 | 1 (most extreme) | 0.04 | 0.20 | 5.000 |
| B | 2 | 2 | 0.10 | 0.15 | 1.500 |
| C | 1 | 3 | 0.16 | 0.32 | 2.000 |
| D | 2 | 4 | 0.25 | 0.20 | 0.800 |
| E | 2 | 5 (least extreme) | 0.45 | 0.13 | 0.2889 |
The probabilities sum to 1 under each hypothesis: under \(H_0\), \(0.04+0.10+0.16+0.25+0.45=1.00\); under \(P_1\), \(0.20+0.15+0.32+0.20+0.13=1.00\). For outcome B, the stagewise tail includes A and B, so its stagewise p-value is \(0.04+0.10=0.14\). Its likelihood ratio is \(0.15/0.10=1.500\). The likelihood-ratio tail includes A, B, and C, since their ratios are \(5.000\), \(1.500\), and \(2.000\). Thus B’s likelihood-ratio p-value is \(0.04+0.10+0.16=0.30\).
For outcome C, the stagewise tail includes A, B, and C, giving \(0.04+0.10+0.16=0.30\). Its likelihood ratio is \(0.32/0.16=2.000\); the likelihood-ratio tail includes A and C, but not B, so its p-value is \(0.04+0.16=0.20\). The two orderings reverse B and C: stagewise ranks B as more extreme, whereas likelihood-ratio ordering ranks C as more extreme. The resulting p-values differ because the tail sets differ, not because the null probabilities have changed.
Why rankings can diverge
The stagewise ordering emphasizes the evidence measure specified for the sequential design. The likelihood-ratio ordering emphasizes relative probability under a chosen alternative and the null. A later-look outcome can have a lower stagewise score than an early-look outcome yet have a larger likelihood ratio, or the reverse. Information accumulated by the stopping look matters to the likelihood ratio, while stagewise ranks need not be determined by that ratio.
A useful comparison is pairwise. For two outcomes \(a\) and \(b\), where \(P_0(a)>0\) and \(P_0(b)>0\), likelihood-ratio ordering ranks \(a\) above \(b\) exactly when \(P_1(a)P_0(b)>P_1(b)P_0(a)\). This cross-multiplied form avoids dividing by a likelihood ratio that could be zero. If the stagewise rule ranks \(b\) above \(a\) but this inequality ranks \(a\) above \(b\), the orderings disagree on that pair. Such a reversal is enough to make their tail sets potentially different; the resulting p-values still require summing or integrating the full null probabilities in each tail.
Worked Example: Likelihood Ratios at Different Information Levels
Suppose a hypothetical trial follows the canonical normal model. At look \(k\), under effect parameter \(\theta\), the standardized statistic satisfies \(Z_k\sim N(\theta\sqrt{I_k},1)\), where \(I_k\) is cumulative information. For a specified alternative \(\theta=0.5\) against \(H_0:\theta=0\), the likelihood ratio at look \(k\), for the observed canonical statistic, is \(L_k=\exp\{0.5\sqrt{I_k}Z_k-(0.5)^2I_k/2\}\). This calculation assumes the canonical normal model and a fixed, hypothesis-independent stopping rule; the full stopped paths must be in the model’s support.
Compare two hypothetical stopped observations. Observation A stops at \(I_1=1\) with \(Z_1=2.1\); observation B stops at \(I_2=2\) with \(Z_2=1.8\). Suppose the stated stagewise rule for this illustration ranks the larger observed \(Z\) as more extreme. It ranks A above B because \(2.1>1.8\).
For A, the log likelihood ratio is \(0.5(1)(2.1)-0.25(1)/2=1.05-0.125=0.925\), so \(L_A=\exp(0.925)\approx2.522\). For B, retaining the unrounded square root in the substitution, \(0.5\sqrt{2}(1.8)-0.25(2)/2\approx1.2728-0.25=1.0228\), and \(L_B=\exp(1.0228)\approx2.781\). Thus the likelihood-ratio ordering ranks B above A, the reverse of the specified stagewise ordering.
These calculations compare the two observed likelihood ratios; they do not by themselves give either ordered p-value. To calculate those p-values, one also needs the null distribution of all stopped outcomes and the ordering of each possible outcome. In particular, the comparison must not treat the two stopping looks as though they were separate, unselected fixed-sample analyses.
The likelihood-ratio ordering depends on the alternative
A stagewise ordering is fixed once its stagewise score and design are specified. A likelihood-ratio ordering can change when the alternative changes. This is a substantive feature: the alternative determines which outcomes count as especially diagnostic of the hypothesized effect, and thus can alter both the ranks and the p-value.
Worked Example: Changing the Specified Effect Changes the Rank
Use the canonical normal model again, with two hypothetical stopped observations: A at \(I_1=1, Z_1=2.0\), and B at \(I_2=2, Z_2=1.7\). Suppose a stagewise rule ranks by larger observed \(Z\), so it ranks A above B. Under the simple alternative \(\theta>0\), the log likelihood ratio for an observation at information \(I\) is \(\ell(\theta)=\theta\sqrt{I}Z-\theta^2I/2\).
At \(\theta=0.5\), A has \(\ell_A=0.5(2.0)-0.25(1)/2=0.875\). B has \(\ell_B=0.5\sqrt{2}(1.7)-0.25(2)/2\approx1.2021-0.25=0.9521\). Therefore \(L_B>L_A\), and this likelihood-ratio ordering ranks B above A.
At \(\theta=1\), A has \(\ell_A=1(2.0)-1(1)/2=1.500\). B has \(\ell_B=1\sqrt{2}(1.7)-1(2)/2\approx2.4042-1=1.4042\). Now \(L_A>L_B\), so the likelihood-ratio ordering ranks A above B. The stagewise ranking has not changed, but the likelihood-ratio ranking has. This illustrates why a reported likelihood-ratio p-value must identify the alternative used; it is not a uniquely determined number from the observed stopped data alone.
Justification and interpretation of ordered p-values
For either ordering, let \(R(y)\) be its evidence score, with larger values defined as more extreme. The p-value is \(p(y)=P_0\{R(Y)\ge R(y)\}\). Under \(H_0\), if the outcome is itself random, the event \(p(Y)\le u\) can contain only outcomes whose ordered null tail probability is at most \(u\). Consequently its null probability is at most \(u\); in a discrete outcome space it can be strictly less because tail probabilities jump over some values. This is the usual super-uniform calibration of an ordered p-value, provided the null stopped-outcome distribution and ordering have been correctly specified.
This calibration does not imply that two valid orderings give the same p-value for observed data. Nor does it imply that an ordering with the smaller observed p-value is universally better. The numerical value is evidence assessed through a particular ordering and a null model; it is not the probability that \(H_0\) is true.
Diagnostics, alternatives, and limitations
When the two results differ, check the construction before interpreting the difference. Confirm that both calculations use the same stopping rule, look schedule, null model, and definition of the stopped outcome. Verify that stagewise ranks use the declared score, and that likelihood ratios use the stated alternative. Finally, recompute each null tail by including every outcome at least as extreme under its own ordering. A tail assembled only from the observed look is generally not the unconditional stopped-outcome p-value.
The main alternative to comparing ordered p-values is to report the likelihood ratio itself or to use a prespecified test statistic and its calibrated null tail. These summaries answer different questions: the likelihood ratio compares relative support for two specified models, while a p-value measures null probability in an ordering-defined tail. Conditional p-values, which condition on the stopping look, are another distinct target; they should not be substituted for unconditional stagewise or likelihood-ratio p-values without stating that change in conditioning.
Both ordered p-values depend on accurate probabilities for the stopped outcomes. In a complex design those probabilities may require integration or recursion, and numerical error can be material in small tails. In continuous models, exact ties often have probability zero; in discrete models, ties can affect tail inclusion and require an explicit convention. The next step in a discrete calculation is to state how tied outcomes are handled, rather than silently omitting or partially counting them.
Common mistake
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the five-outcome example, calculate the stagewise and likelihood-ratio p-values for outcome D.
- Why can an outcome with a smaller observed \(Z\) have a larger likelihood ratio at a later look?
- What additional specification is needed before a likelihood-ratio ordering is fully defined?
- If two orderings give different p-values, does that alone show one calculation is invalid? Explain.
Solutions. (1) D has stagewise rank fourth, so its stagewise tail includes A, B, C, and D: \(0.04+0.10+0.16+0.25=0.55\). Its likelihood ratio is \(0.20/0.25=0.800\); outcomes with ratio at least \(0.800\) are A, B, C, and D, also giving \(0.55\). (2) The likelihood ratio includes both the observed statistic and cumulative information. At a later look the information contribution can outweigh a lower \(Z\), as the canonical-model calculation demonstrates. (3) Specify the alternative distribution or effect value, as well as the null model and stopped-outcome probabilities. (4) No. If each ordering and its null tail are correctly specified, different values can result simply because they include different outcomes in their tails.