Reporting the Ordering Behind a Sequential P-Value

Why the Ordering Must Be Reported

A sequential p-value is not determined by the observed test statistic alone. After early stopping, the data include both the evidence observed and the rule that determined when observation ended. A p-value also depends on how stopped outcomes are ordered from less to more extreme. Two analysts can use the same trial data and null hypothesis but obtain different valid p-values if they use different orderings or condition on different events.

The earlier material on stagewise and likelihood-ratio p-values develops important choices of ordering. Here the practical task is to make the choice explicit enough that another analyst can reconstruct the reported probability. This means stating not only the final number, but the stopping rule, null law, ordered object, tie treatment, and computational procedure.

Definition: Let $W$ denote the complete stopped outcome: the stopping look and all data observed through that look. Let $W^*$ be a replicate stopped outcome generated under the specified null model and stopping rule. If $q(W)$ is a declared ordering score, with larger values defined as more extreme, the inclusive-tail p-value is $$ p = P_0\{q(W^*) \ge q(w_{\mathrm{obs}})\}. $$ Here $w_{\mathrm{obs}}$ is the observed stopped outcome. The definition includes outcomes tied with the observed score. A conditional version instead calculates this probability under the null distribution given a specified event, such as the observed stopping look.

The score $q$ can be a stagewise statistic, a likelihood ratio, or another prespecified measure of extremeness. There is no universal ordering that applies to every sequential test. Consequently, “the sequential p-value” is incomplete as a report unless the ordering and probability distribution are identified.

What a Reproducible Report Specifies

A reader should be able to reconstruct the sample space of stopped outcomes and determine which of those outcomes count in the p-value tail. In a group sequential trial, report the planned looks or information times, the boundaries and other stopping conditions, and the rule for what happens if no boundary is crossed. State whether the probability is unconditional under the full design or conditional on a stopping event.

Also identify the null hypothesis and the null distribution of the interim statistics, including any dependence across looks. For example, in a canonical Gaussian model with information levels $I_1 < \cdots < I_K$, the null joint distribution is commonly specified by $$ (Z_1,\ldots,Z_K) \sim N_K(0,\Sigma), \qquad \Sigma_{ij} = \sqrt{\frac{I_{\min(i,j)}}{I_{\max(i,j)}}}. $$ This states the assumed joint law under $H_0$; the stopping rule then determines which portions of that vector become observed stopped outcomes. If the analysis uses a different null model, different information times, or a different covariance structure, those details must be reported instead.

The ordering needs equal care. Give its mathematical definition, say whether larger or smaller values are more extreme, and explain how it treats outcomes from different stopping looks. If the ordering is conditional on a stopping look or uses a likelihood ratio, state that explicitly and identify any alternative parameter used to form the likelihood ratio. For discrete statistics, say whether tied outcomes are included in full, excluded, or assigned a fractional weight. The conventional inclusive-tail definition above includes ties.

Finally, describe how the probability was computed. For enumeration, identify the outcome set and probability weights. For numerical integration, report the integrand or event, integration region, method, tolerances, and software or algorithm sufficiently clearly to reproduce the calculation. For simulation, provide the number of replicates, random-number generator and seed, and the estimator and its Monte Carlo uncertainty. A rounded p-value alone cannot reveal whether the tail was evaluated correctly or accurately.

Deriving the Tail Calculation

The definition gives a direct recipe. Under $H_0$, generate or enumerate stopped outcomes using the stated stopping rule. Retain each outcome whose score is at least as extreme as the observed score, then add its null probability. This is not the same as calculating a fixed-look tail probability from the statistic at the observed look: paths that stop at other looks may also belong to the ordered tail.

p = \sum_{w \in \mathcal{W}} \mathbf{1}\{q(w) \ge q(w_{\mathrm{obs}})\}P_0(W=w),

where $\mathcal{W}$ is the set of possible stopped outcomes and $\mathbf{1}\{\cdot\}$ is one when its condition holds and zero otherwise. For a continuous outcome, replace the sum by an integral over the stopped-outcome region defined by $q(w) \ge q(w_{\mathrm{obs}})$. For a conditional p-value given event $C$, use the conditional weights $P_0(W=w\mid C)$ for outcomes in $C$, rather than the unconditional weights. This derivation also explains why stating only the test statistic is insufficient: without the outcome space, null weights, and tail rule, the sum or integral is not defined.

Worked Example: Exact Tail Probability With a Boundary-Induced Tie

Consider a hypothetical two-look test under $H_0$ in which $X_1$ and $X_2$ are independent Bernoulli random variables with success probability $0.5$. The trial stops at look 1 if $X_1=1$; otherwise, it continues and stops after observing $X_2$. Define $q$ as the total number of successes observed by stopping. Suppose the observed outcome is $A=(J=1,X_1=1)$.

The stopping rule creates three possible stopped outcomes. Their null probabilities follow from independence:

\begin{aligned} P_0(A) &= P_0(X_1=1)=0.5,\\ P_0(B) &= P_0(X_1=0,X_2=0)=0.5(0.5)=0.25,\\ P_0(C) &= P_0(X_1=0,X_2=1)=0.5(0.5)=0.25. \end{aligned}

Their scores are $q(A)=1$, $q(B)=0$, and $q(C)=1$. Since the observed score is 1, the inclusive tail includes both $A$ and the tied outcome $C$:

p = P_0\{q(W^*) \ge 1\} = P_0(A)+P_0(C) = 0.50+0.25 = 0.75.

Thus the reported p-value is 0.75 under this explicitly defined ordering and unconditional null distribution. Excluding ties would instead give $P_0\{q(W^*)>1\}=0$; that is a different tail convention, not a rounding variation. The mid-p convention would give $P_0(q>1)+\tfrac12P_0(q=1)=0+\tfrac12(0.75)=0.375$. A report must say which convention it uses.

Conditional and Unconditional Probabilities Are Different Reports

Conditioning on the stopping look changes the null distribution of the stopped outcome. That change can be appropriate for a specified conditional p-value, but it must not be introduced silently. The next example uses the same hypothetical design to isolate the distinction.

Worked Example: Conditioning on the Observed Stopping Look

Suppose the observed stopped outcome is $C=(J=2,X_1=0,X_2=1)$. It has score $q(C)=1$. For the unconditional score ordering, the outcomes at least as extreme are $A$ and $C$, so the calculation is

p_{\mathrm{unconditional}} = P_0(A)+P_0(C) = 0.50+0.25 = 0.75.

Now define a different p-value conditional on stopping at look 2. Under the stopping rule, $J=2$ means $X_1=0$. Conditional on this event, the independent $X_2$ is still Bernoulli with success probability 0.5. The possible outcomes are therefore $B$ and $C$, each with conditional probability $0.5$. Only $C$ has score at least 1, giving

p_{\mathrm{conditional}} = P_0\{q(W^*) \ge 1 \mid J=2\} = 0.5.

Both calculations are reproducible, but they answer different questions because their reference distributions differ. The report must label the result as unconditional or conditional, and specify the conditioning event precisely.

Different Orderings Can Give Different Answers

Even with the null law, stopping rule, and observed outcome held fixed, changing the ordering can change the tail probability. A likelihood-ratio ordering is one standard choice when a particular alternative is of interest: rank outcomes by their likelihood under that alternative divided by their likelihood under $H_0$, with larger ratios treated as more extreme. That choice depends on the specified alternative and is not interchangeable with an ordering based simply on the number of observed successes.

Worked Example: Score Ordering Versus Likelihood-Ratio Ordering

Use the three stopped outcomes $A,B,C$ and their null probabilities from the first example. The observed outcome is $A$. Compare the success-count score $q$ with a likelihood-ratio ordering for the simple alternative $P(X_i=1)=0.8$. Under that alternative, the stopped-outcome probabilities are

P_{0.8}(A)=0.8,\qquad P_{0.8}(B)=(0.2)^2=0.04,\qquad P_{0.8}(C)=(0.2)(0.8)=0.16.

Dividing each alternative probability by its null probability gives likelihood ratios $1.6$ for $A$, $0.16$ for $B$, and $0.64$ for $C$. For observed $A$, the likelihood-ratio tail includes only $A$, so

p_{\mathrm{LR}}=P_0\{LR(W^*) \ge 1.6\}=P_0(A)=0.50.

The success-count ordering instead ties $A$ and $C$, both with one success, giving $p_q=P_0(A)+P_0(C)=0.50+0.25=0.75$. The difference is not a calculation error: the orderings define different sets of outcomes as at least as extreme. A reproducible report states which ordering was chosen and, for the likelihood-ratio version, that the alternative parameter was 0.8.

Computational Checks, Alternatives, and Limitations

The computational method should follow the probability model and the size of the stopped-outcome space. If the possible outcomes are few, exact enumeration makes the tail membership and probability weights easy to audit. With continuous test statistics, numerical integration can evaluate the null probability of the ordering region. Simulation can approximate that probability when analytic or numerical integration is difficult, but should include a measure of Monte Carlo uncertainty; it does not turn a poorly specified ordering into a well-defined p-value.

As a practical check, independently verify that the stopping rule partitions the possible paths correctly, that the null probabilities sum to one, and that every included outcome satisfies the reported ordering inequality. For a conditional calculation, verify that the conditional probabilities over the restricted outcome space sum to one. With numerical integration or simulation, check stability against tighter tolerances or a larger simulation run. In particular, small tail probabilities require enough numerical precision or simulation replicates to support the reported digits.

The main alternatives answer different inferential specifications. An unconditional p-value uses the full null distribution of stopped outcomes. A stopping-look-conditional p-value restricts that distribution to outcomes with the observed stopping look. A stagewise ordering ranks outcomes according to a declared stage-specific convention, while a likelihood-ratio ordering ranks them by evidence against the null relative to a specified alternative. Neither conditioning choice nor ordering is universally best; the intended inferential procedure determines which is appropriate.

These p-values also have limits. The value is a tail probability under the stated null model and analysis rule; it is not the probability that $H_0$ is true. A rounded result does not establish that the assumptions, stopping rule, or ordering were appropriate. If the actual look timing differs from the plan, or the implemented stopping rule differs from the reported one, the reference distribution may no longer describe the analysis that was performed. Reporting detail makes such differences detectable, but does not by itself correct a misspecified design or model.

Common Reporting Pitfall

A common mistake is to report, for example, “the sequential p-value was 0.03,” then justify it only by naming the observed boundary crossing or the test statistic. This does not say which stopped paths were counted as at least as extreme, whether ties were included, or whether the calculation was conditional on the observed look. Another mistake is to assume that two standard orderings must produce the same p-value.

The correction is to report the probability statement itself: identify the observed stopped outcome, the null distribution and stopping rule, the ordering score and its direction, and the tie convention. Then identify the computational procedure and its precision. A reader should be able to determine the tail event before seeing its numerical probability.

Key takeaway: A sequential p-value is reproducible only when its reference distribution and ordering are reproducible. Report the stopping rule, null model, unconditional or conditional target, ordering, tie treatment, and computational method alongside the result.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In the first worked example, why is outcome $C$ included in the inclusive tail for observed $A$?
What distinction does the value 0.75 versus 0.50 illustrate in the conditional-look example?
For the likelihood-ratio ordering in the final worked example, which stopped outcomes are at least as extreme as observed $A$, and what is their null probability?
Name four elements, in addition to the numerical p-value, that should be reported to make the result reproducible.
Why is a mid-p value not the same as the inclusive-tail p-value when the statistic is discrete?

Solutions. (1) $C$ has the same score as $A$, so the inclusive rule $q(W^*) \ge q(A)$ counts it as tied; its null probability is 0.25. (2) The values are respectively unconditional and conditional on $J=2$; their reference distributions differ. (3) Only $A$ has likelihood ratio at least 1.6, and its null probability is 0.50. (4) Suitable elements include the stopping rule, null distribution, conditioning target, ordering and direction, tie rule, and computational method. (5) The inclusive tail counts the full probability mass tied with the observed statistic, whereas the mid-p assigns half of that mass.