Calibration Across Different Look Schedules

Why the Look Schedule Is Part of Calibration

A group sequential p-value describes an outcome from a procedure that could have stopped at several planned looks. Its null reference distribution therefore concerns the whole stopping procedure, not just the statistic at the look where observation ended. The planned timing of the looks affects the joint distribution of the statistics and, consequently, the probability assigned to outcomes deemed at least as extreme as the observed one.

The key distinction is between using the null distribution for the schedule that was actually prespecified and carrying a p-value calibration across to a different schedule. The first is the basis for validity. The second need not preserve the same null distribution, even if the test statistic and stopping boundary have the same form.

Definition: Let $O_S$ denote the complete stopped outcome under a prespecified schedule $S$, including the stopping look and the statistic observed there. Let $R_S(O_S)$ be a prespecified ordering score, with larger scores defined to be more extreme against $H_0$. The nonrandomized stagewise p-value is $$ p_S(o)=P_{0,S}\{R_S(O_S)\ge R_S(o)\}, $$ where $P_{0,S}$ is probability under $H_0$ and the complete design for schedule $S$, including its stopping rule. The ordering and the distribution used to calculate the tail probability are both part of this definition.

The notation $p_S$ emphasizes that the reference distribution belongs to a schedule. It does not mean that every reported p-value must be recomputed for every imaginable schedule: it means that its validity argument uses the prespecified schedule that generated the outcome. Earlier material on stagewise p-values establishes the role of the ordering; here the focus is how the schedule affects its calibration.

A Two-Look Model and Its Schedule-Specific Tail

Consider a one-sided test with two planned looks. Under $H_0$, the standardized statistics have the canonical joint normal model $$ (Z_1,Z_2)\sim N\left( \begin{pmatrix}0\\0\end{pmatrix}, \begin{pmatrix}1&\rho\\\rho&1\end{pmatrix} \right), \qquad \rho=\sqrt{I_1/I_2}, $$ where $I_1<I_2$ are the information levels. Thus each statistic has a standard normal marginal distribution, and the correlation depends on the information fraction at the first look. The schedule and information levels are fixed in advance; the null model is correctly specified; and the statistics are not altered by unplanned selection.

Use the following stopping rule and ordering. Stop for efficacy at look 1 if $Z_1\ge0$; otherwise continue to look 2. Rank every early efficacy stop ahead of every final-look outcome, and order outcomes within each look by decreasing $Z$. There are no ties with positive probability in this continuous model. For a final-look outcome with $Z_2=z$, the tail includes an early stop at any statistic and a final outcome at least as large as $z$:

p_\rho(z) =P_0(Z_1\ge0)+P_0(Z_1<0,\ Z_2\ge z).

This tail is not just $P_0(Z_2\ge z)$. That latter probability ignores the early outcomes that the stated ordering places ahead of all final outcomes.

Worked Example: A Final-Look Outcome Under Schedule A

Suppose schedule A has $I_1/I_2=0.25$, so $\rho_A=0.5$. A hypothetical trial continues to the final look and observes $Z_2=0$. Calculate its stagewise p-value under the specified ordering.

The early-stop probability is $P(Z_1\ge0)=0.5$. For a standard bivariate normal pair with correlation $\rho$, $$ P(Z_1<0,\ Z_2\ge0)=\frac14-\frac{\arcsin(\rho)}{2\pi}. $$ Consequently, $$ p_A(0)=\frac12+\frac14-\frac{\arcsin(0.5)}{2\pi} =\frac34-\frac{\pi/6}{2\pi} =\frac34-\frac1{12} =\frac23\approx0.6667. $$

The exact p-value is $2/3$; $0.6667$ is its rounded representation. The calculation uses the final statistic together with the probability of an earlier efficacy stop, as required by the ordering.

How Changing the Schedule Changes the Null Distribution

Keep the stopping boundary and ordering fixed, but use schedule B with $I_1/I_2=0.81$, giving $\rho_B=0.9$. For the same final-look value $z=0$, the early-stop probability remains $0.5$, but the probability of continuing and then having $Z_2\ge0$ changes with the correlation. The same quadrant identity gives

p_B(0)=\frac34-\frac{\arcsin(0.9)}{2\pi} \approx0.5718.

Thus the same observed final statistic, under the same ordering and nominal boundary, has different schedule-specific tail probabilities: $p_A(0)=2/3$ and $p_B(0)\approx0.5718$. The reason is not that either schedule changes the marginal null distribution of $Z_2$. Instead, it changes how $Z_1$ and $Z_2$ co-vary, and therefore changes the probability of reaching the final look with a statistic in the specified tail.

It is useful to state the cutoff precisely. For final-look outcomes, $p_A(z)$ decreases with $z$, so $$ p_A(z)\le p_A(0)\quad\Longleftrightarrow\quad z\ge0. $$ This equivalence uses the exact cutoff $p_A(0)=2/3$, not the rounded number $0.6667$. Under schedule B, the probability of this tail among outcomes that reach the final look is $P_{0,B}(Z_1<0,Z_2\ge0)$. For the complete stopped outcome, the ordering also includes every early efficacy stop, so $$ P_{0,B}\{p_A(O_B)\le p_A(0)\} =P_{0,B}(Z_1\ge0)+P_{0,B}(Z_1<0,\ Z_2\ge0) =p_B(0)\approx0.5718. $$ The corresponding complete-outcome event under schedule A has probability $2/3$. In particular, replacing the exact cutoff with $0.6667$ and asserting an exact event equality is not justified: a rounded value can correspond to a slightly different cutoff.

Worked Example: A Different Schedule Can Be Anti-Conservative

To see why changing schedules can affect validity, use schedule C with the same two-look stopping rule and ordering, but $I_1/I_2=0.01$, so $\rho_C=0.1$. Consider applying schedule A's p-value mapping to outcomes from schedule C and using the exact threshold $q=p_A(0)=2/3$.

For a final outcome, $p_A(z)\le2/3$ holds exactly when $z\ge0$. Including the early stops, the probability of this event under schedule C is $$ P_{0,C}\{p_A(O_C)\le2/3\} =P(Z_1\ge0)+P(Z_1<0,\ Z_2\ge0) =\frac34-\frac{\arcsin(0.1)}{2\pi} \approx0.7341. $$

Since $0.7341>2/3$, the carried-over p-value is anti-conservative at this threshold: its null probability of being at most $q$ exceeds $q$. The schedule C-specific p-value instead uses $\rho_C=0.1$ in its null tail calculation. This example shows that cross-schedule calibration can fail, not merely that the numerical p-value can change.

Why the Prespecified Design Gives a Valid P-Value

Theorem: If the null distribution of the complete stopped outcome is correctly specified for a fixed schedule $S$, and the ordering is fixed in advance, then the nonrandomized tail p-value $p_S(O_S)$ is super-uniform under $H_0$: $$ P_{0,S}\{p_S(O_S)\le q\}\le q\qquad(0\le q\le1). $$ If the ordered outcome has a continuous distribution with no positive-probability ties, the inequality is an equality: $p_S(O_S)\sim\operatorname{Uniform}(0,1)$.

The result is a rank argument. Order possible outcomes from most to least extreme under the fixed ordering. For any threshold $q$, the set of outcomes whose null tail probability is at most $q$ is an upper tail of that ordering. Its null probability cannot exceed $q$, because the tail probability of its least-extreme member is the probability assigned to that member and all outcomes ahead of it. With no ties and a continuous distribution, the tail probability itself is the probability-integral transform applied to the ordered outcome, which is uniform.

For the two-look example, the same argument can be seen directly. Early outcomes have p-values $P(Z_1\ge Z_1^{\mathrm{obs}})$, which occupy the interval from 0 to $0.5$. Their unconditional probability is $0.5$, and the continuous normal tail transform spreads that probability uniformly over this interval. Final outcomes have p-values $0.5+P(Z_1<0,Z_2\ge Z_2^{\mathrm{obs}})$, which occupy the interval from $0.5$ to 1. Conditional on continuing, the continuous conditional distribution of $Z_2$ gives a uniform tail rank over this interval; the interval has probability mass $0.5$. Together, the two pieces give a uniform p-value under the fixed schedule.

If the ordered outcome is discrete, the tail probability can jump. The nonrandomized p-value is then generally super-uniform rather than exactly uniform: the probability of falling below a cutoff is no greater than that cutoff. Randomization within tied outcomes can recover exact uniformity under the null, but it is not necessary for the super-uniform validity statement.

Worked Example: Calibrating a Threshold Under the Matching Schedule

Return to schedule A and the exact threshold $q=2/3$. For a complete stopped outcome, $p_A(O_A)\le2/3$ includes early efficacy stops and final outcomes with $Z_2\ge0$. Its null probability is $$ P_{0,A}\{p_A(O_A)\le2/3\} =P(Z_1\ge0)+P(Z_1<0,\ Z_2\ge0) =\frac34-\frac{\arcsin(0.5)}{2\pi} =\frac23. $$

At this particular threshold the continuous, matching-schedule p-value has exact calibration. This does not mean that $2/3$ is a conventional significance level; it is a transparent point at which to check the null distribution. The theorem applies at every $q$ between 0 and 1, not only at this example's threshold.

Diagnostics, Alternatives, and Limitations

When checking a reported stagewise p-value, first confirm that its null distribution matches the planned information levels, number of looks, stopping rule, and ordering. Then check that the probability calculation includes all outcomes ranked at least as extreme as the observation, including outcomes at other stopping looks where the ordering requires them. A discrepancy between a reported p-value and the matching schedule's tail probability can indicate an omitted stopping path, an incorrect correlation, or a mismatch in the ordering.

The main alternatives answer different questions. A fixed-final-look p-value uses the final statistic's marginal null distribution and does not generally account for early stopping; it is not a substitute for a stopped-outcome p-value when the ordering concerns the entire procedure. A conditional p-value calculates a tail probability given the stopping look, so it describes a conditional reference distribution rather than the unconditional stopped-outcome ordering used here. A schedule-specific stagewise p-value includes the stopping process in its reference distribution. These quantities may differ without any arithmetic error because they rank outcomes or condition on information differently.

Calibration also depends on the assumptions behind the null model. If the information levels are not those used in the calculation, the canonical correlation may be wrong. If the schedule or ordering is chosen after examining results, the stated rank argument does not automatically apply to the selected design: the selection mechanism must also be included in the null distribution, or a valid adjustment must be made. Finally, a valid null p-value does not by itself establish that the assumed model is scientifically appropriate or that the ordering is the only reasonable one.

Pitfall: “The final statistic is standard normal under the null, so its p-value calibration is unchanged by the look schedule.”
Correction: A standard normal marginal for $Z_2$ does not determine the null distribution of the stopped outcome. The joint distribution across looks and the stopping rule determine which final outcomes are observed and how the chosen ordering ranks them.
Pitfall: “A p-value calculated under one prespecified schedule remains valid if the schedule is replaced by another one.”
Correction: Validity follows from the null tail probability for the schedule and ordering used. Cross-schedule use requires a separate justification and can be anti-conservative.

Key takeaway: Fix the schedule, stopping rule, and ordering before examining the data, then calculate the p-value from the null distribution of the complete stopped outcome under that design. Changing the look schedule changes that reference distribution and can change, or invalidate, the calibration.

Check Your Understanding

Answer each question using the distinction between a schedule-specific null tail and a carried-over calibration.

In the two-look model, what feature of the look schedule changes the correlation between $Z_1$ and $Z_2$?
For schedule A, why is the stagewise p-value for a final outcome $Z_2=0$ not just $P(Z_2\ge0)$?
Suppose a schedule-specific p-value has a discrete null distribution and ties are not randomized. What validity statement applies?
Why can choosing a schedule after examining interim results invalidate the fixed-schedule rank argument?

Solutions. (1) The information fraction $I_1/I_2$ determines $\rho=\sqrt{I_1/I_2}$. (2) The stated ordering ranks every early efficacy stop ahead of every final outcome, so the tail includes both $Z_1\ge0$ and $Z_1<0,\ Z_2\ge0$. (3) The p-value is super-uniform: $P_0(p\le q)\le q$; it need not be exactly uniform because of jumps at ties. (4) The selected schedule is then data-dependent, so its null reference distribution is not necessarily the distribution for a schedule fixed in advance; validity must account for the selection process or use an appropriate adjustment.