Deriving a Stagewise P-Value From the Null Distribution

From a Stopped Outcome to a Null Tail Probability

When a group sequential trial stops early, its observed result is not just a test statistic at one look. It includes which look was reached and the statistic observed there. A stagewise p-value uses that complete stopped outcome: it compares the observation with all outcomes ranked at least as strongly by a prespecified ordering, then calculates the probability of that set under $H_0$.

The key calculation is a probability over the stopped-outcome space, not an ordinary tail area for the statistic at the observed look alone. The stopping rule determines which outcomes can occur at each look, while the joint null distribution determines their probabilities.

Definition: Let $K$ be the maximum number of scheduled looks, and let $O=(J,Z_J)$ denote the observed outcome, where $J$ is the stopping look and $Z_J$ is the statistic at that look. Let $r(j,z)$ be a prespecified evidence score, with larger values meaning stronger evidence against $H_0$. For an observed outcome $o=(j,z)$, its stagewise p-value for this ordering is $$ p_{\mathrm{st}}(o)=P_0\{r(J,Z_J)\ge r(j,z)\}. $$ The probability includes every possible stopping look. The weak inequality includes outcomes tied with the observation.

The ordering is part of the definition. Different evidence orderings can give different p-values for the same stopped outcome. In this tutorial, examples use an ordering by the observed statistic itself, $r(j,z)=z$, so larger values mean stronger evidence. This is an illustrative ordering, not a claim that every application should rank outcomes that way.

Representing the Stopping Regions

Suppose the stopping rule is fixed in advance. For each look $j$, define $D_j$ as the set of statistic histories $(z_1,\ldots,z_j)$ for which the trial first stops at look $j$. These regions are disjoint: one history cannot first stop at two different looks. Their union is the set of possible stopped histories.

Let $f_{0,j}(z_1,\ldots,z_j)$ be the joint null density of the statistics through look $j$. For a discrete model, use the corresponding joint probability mass function instead. If the observed score is $t=r(j_{\mathrm{obs}},z_{\mathrm{obs}})$, the event “ranked at least as extreme” consists of histories in $D_j$ whose stopping statistic has score at least $t$. Therefore,

p_{\mathrm{st}}(o) = \sum_{j=1}^{K} \int_{D_j\cap\{r(j,z_j)\ge t\}} f_{0,j}(z_1,\ldots,z_j)\, dz_1\cdots dz_j .

For discrete outcomes, replace each integral by a sum of joint null probabilities over the qualifying histories. This expression is the derivation: partition the ranked tail event by the mutually exclusive stopping look, and calculate each part under the joint null law. No independence assumption is needed for the partition. The density must, however, be the correct joint density for the process being analyzed.

In the common canonical normal model under $H_0$, the standardized statistics have mean zero, unit variance, and correlation $\operatorname{Corr}(Z_i,Z_j)=\sqrt{I_i/I_j}$ for $i<j$, where $I_i$ is the information at look $i$. This model presumes independent increments and the usual normal or adequately accurate normal approximation. The formula above then uses the appropriate multivariate normal density, restricted to each stopping region. A different design or statistic may require a different joint null model.

Worked Examples

Worked Example: Summing a Discrete Joint Null Distribution

Consider a hypothetical two-look process with the following joint null distribution. At look 1, stop if $Z_1=2$; otherwise continue, in which case $Z_1=0$. The joint probabilities are $P_0(Z_1=2,Z_2=0)=0.10$, $P_0(Z_1=2,Z_2=1)=0.10$, $P_0(Z_1=0,Z_2=0)=0.40$, and $P_0(Z_1=0,Z_2=1)=0.40$. Use $r(j,z)=z$, and suppose the observed outcome is a final-look result $Z_2=1$.

The observed score is $t=1$. At look 1, the stopped outcomes with score at least 1 have $Z_1=2$; their total probability is $0.10+0.10=0.20$. At look 2, the process continued only when $Z_1=0$, and the qualifying final outcome has $Z_2=1$, with probability $0.40$. Thus

p_{\mathrm{st}}=0.20+0.40=0.60.

The calculation uses joint probabilities and respects the stopping rule. The $Z_2$ values paired with $Z_1=2$ do not represent observed final-look outcomes: the trial stopped before observing them.

Worked Example: A Correlated Normal Process With a Simple Boundary

For a tractable hypothetical calculation, consider two canonical normal statistics under $H_0$, with information ratio $I_1/I_2=1/2$. Thus $Z_1$ and $Z_2$ are standard normal and have correlation $\rho=\sqrt{1/2}$. Stop at look 1 when $Z_1\ge 0$; otherwise continue to look 2. Use $r(j,z)=z$, and consider a final-look observation $Z_2=0$.

The observed score is zero. The qualifying stopped outcomes are (i) look-1 outcomes with $Z_1\ge0$, and (ii) look-2 outcomes with $Z_1<0$ and $Z_2\ge0$. Hence

p_{\mathrm{st}} =P_0(Z_1\ge0)+P_0(Z_1<0,Z_2\ge0).

For a standard bivariate normal pair with correlation $\rho$, symmetry gives $P_0(Z_1<0,Z_2\ge0)=1/4-\arcsin(\rho)/(2\pi)$. Here $\arcsin(1/\sqrt{2})=\pi/4$, so the second probability is $1/4-1/8=1/8$. Also, $P_0(Z_1\ge0)=1/2$. Therefore,

p_{\mathrm{st}}=\frac{1}{2}+\frac{1}{8}=0.625.

The stopping regions are disjoint, and the dependence between the looks enters through the bivariate probability. The zero boundary is chosen for transparent arithmetic; this example is not a recommendation for a clinical trial boundary.

Worked Example: How Independence Changes the Calculation

Now consider a different hypothetical process whose two stage-specific statistics $X_1$ and $X_2$ are independent $N(0,1)$ under $H_0$. Stop at look 1 if $X_1\ge2$; otherwise observe $X_2$ at look 2. This is an independent-statistics model, not the canonical cumulative-statistic model. Use the score $r(j,x)=x$, and suppose the observed final statistic is $X_2=1.5$.

The score threshold is 1.5. The qualifying look-1 outcomes have $X_1\ge2$, whose probability is $1-\Phi(2)$. The qualifying look-2 outcomes require both continuation, $X_1<2$, and $X_2\ge1.5$. Independence gives

p_{\mathrm{st}} = [1-\Phi(2)]+\Phi(2)[1-\Phi(1.5)].

Using $\Phi(2)=0.97725$ and $\Phi(1.5)=0.93319$, the arithmetic is $0.02275+(0.97725)(0.06681)=0.08804$, to four significant figures. The look-1 tail is included because those stopped outcomes also meet the score threshold. The factor $\Phi(2)$ accounts for the requirement that a look-2 outcome was actually observed.

Assumptions, Checks, and Common Errors

A stagewise calculation is only as appropriate as its inputs. Before evaluating the tail probability, verify that the stopping rule and look schedule are represented correctly, that the ordering is specified before evaluating the observation, and that the joint null distribution matches the statistics and design. In a normal approximation, check the covariance structure and any approximation assumptions; in numerical integration, also check that probability mass has not been lost at region boundaries or in the tails.

Using only the observed-look tail. Calculating $P_0(Z_j\ge z_{\mathrm{obs}})$ ignores qualifying stopped outcomes at other looks and may include histories that would have stopped earlier.
Ignoring dependence. Multiplying marginal probabilities is justified only under the relevant independence assumptions. Canonical cumulative statistics across looks are generally correlated.
Applying the stopping rule after the fact. Each term must be restricted to histories that first stop at that look. Future statistics after a stop are not part of the observed outcome.
Changing the ordering without notice. The ranked tail depends on $r$. A calculation under one ordering is not automatically the p-value under another.

The main alternative is an ordinary fixed-look tail probability, which refers only to a statistic at a specified look and does not, by itself, rank the full stopped-outcome space. Conditioning on the observed stopping look would also define a different probability calculation: it restricts attention to outcomes at that look rather than summing over all qualifying stopping looks. These quantities answer different questions and should not be substituted silently for the stagewise definition.

The formula is general, but its practical limits are important. The joint null distribution or its approximation must be credible, the stopping regions must be encoded accurately, and the chosen ordering must be meaningful for the scientific question. Complex boundaries may require numerical integration or simulation; numerical results should be checked against probability-mass accounting and an independent calculation where feasible. The definition alone does not establish any broader error-rate property of the resulting p-value.

Key takeaway: Derive a stagewise p-value by taking the null probability of the entire set of stopped outcomes ranked at least as extreme as the observation. Partition that set by stopping look, enforce each look’s stopping region, and use the joint null distribution.

Check Your Understanding

For each question, identify the relevant event before calculating or interpreting its probability.

Why is the stagewise tail probability divided into terms indexed by stopping look?
In the independent-normal example, what event contributes the look-2 term?
Suppose the observed score is $t$, and ties are possible. Should the ranked tail use $r>t$ or $r\ge t$?
Why can marginal normal tails at two looks not generally be multiplied for canonical cumulative statistics?
What must change in the integral formula if the null distribution is discrete?

Solutions. (1) The stopping-look regions are disjoint and together partition the stopped outcomes, so summing their probabilities counts the full ranked tail once. (2) It is $X_1<2$ and $X_2\ge1.5$: the process must continue, and the final score must meet the threshold. (3) Use $r\ge t$, so outcomes tied with the observation are included. (4) Canonical statistics have nonzero correlation, so joint event probabilities do not generally factor into products of marginal probabilities. (5) Replace each density integral by a sum of joint null masses over qualifying stopped histories.

Deriving a Stagewise P-Value From the Null Distribution

What You'll Learn