Stagewise P-Values With Unequal Information Increments

Why Unequal Information Increments Matter

Interim looks are not always separated by equal amounts of statistical information. A trial might review results after 20%, 60%, and 100% of the planned information has accrued, or the timing might be irregular because information depends on observed event counts. The Z-statistics at these looks are correlated, and their correlations depend on the information accrued at each look. If a stagewise p-value is calculated using the wrong correlations, the probability assigned to the observed stopped outcome can be wrong even when the observed Z-statistic and stopping boundaries are recorded correctly.

This tutorial considers a one-sided efficacy test with no futility boundary. It focuses on calculating stagewise p-values using the actual information levels and the corresponding canonical joint normal distribution. The ordering must be specified: here, as in the one-sided stagewise ordering, efficacy stops at earlier looks rank as more extreme than outcomes at later looks, and outcomes at the same look are ordered by their Z-statistic. Different orderings can give different p-values for the same observed data.

Definition: Let $I_1<I_2<\cdots<I_K$ be the cumulative information at the planned looks, and let $Z_j$ be the standardized test statistic at look $j$. Under the canonical model at the null boundary, the vector $(Z_1,\ldots,Z_K)$ has a multivariate normal distribution with mean zero, unit variances, and $$ \operatorname{Corr}_0(Z_i,Z_j)=\sqrt{\frac{I_i}{I_j}},\qquad i<j. $$ Equivalently, $\operatorname{Cov}_0(Z_i,Z_j)=\sqrt{I_i/I_j}$. The information increments are $I_1$ and $I_j-I_{j-1}$ for $j\geq2$; they need not be equal.

Model, Stopping Rule, and Stagewise Tail

The canonical model applies when the sequential score has independent increments and its variance is proportional to information. For example, with a normally distributed score process $S(I)$, under the null $S(I)$ has mean zero, variance $I$, and independent increments over disjoint information intervals. The standardized statistic is $Z_j=S(I_j)/\sqrt{I_j}$. The conclusions below also apply to settings where the canonical joint normal model is a suitable approximation. The null distribution is evaluated at the least-favourable boundary of a one-sided null hypothesis, $H_0:\theta\leq0$, versus $H_a:\theta>0$, in a model with nonnegative drift under $\theta>0$.

Let $b_j$ be the upper efficacy boundary at look $j$. An efficacy stop at look $j$ occurs when $Z_j\geq b_j$, provided there was no earlier crossing. Thus its event is $E_j=\{Z_1<b_1,\ldots,Z_{j-1}<b_{j-1},Z_j\geq b_j\}$. Suppose the observed trial first crosses its boundary at look $k$, with observed statistic $z_k\geq b_k$. Under the stated stagewise ordering, all efficacy stops before look $k$ are more extreme than the observed outcome; at look $k$, outcomes with $Z_k\geq z_k$ are at least as extreme. Later-look stops are not included in this ordered tail.

Stagewise p-value: For an observed efficacy stop at look $k$, the one-sided stagewise p-value under this ordering is $$ p_{\mathrm{stage}}(k,z_k) = \sum_{j=1}^{k-1}P_0(E_j) + P_0(Z_1<b_1,\ldots,Z_{k-1}<b_{k-1},Z_k\geq z_k). $$ For $k=1$, the sum is empty and the expression reduces to $P_0(Z_1\geq z_1)$. For later looks, both the earlier stopping probabilities and the joint probability of reaching look $k$ and attaining at least the observed statistic matter.

The joint probability in the final term is not generally the marginal normal tail $P_0(Z_k\geq z_k)$. That marginal tail includes paths that crossed an efficacy boundary earlier and therefore would not have been observed at look $k$. The stagewise calculation respects the stopping rule by restricting the probability to paths with no earlier crossing.

Deriving the Unequal-Look Dependence

The covariance formula follows directly from independent score increments. For $i<j$, write $S(I_j)=S(I_i)+\{S(I_j)-S(I_i)\}$. The increment in braces is independent of $S(I_i)$. Consequently, the covariance of the two unstandardized scores is $I_i$, while their standard deviations are $\sqrt{I_i}$ and $\sqrt{I_j}$. Standardizing gives

\operatorname{Corr}_0(Z_i,Z_j) = \frac{\operatorname{Cov}_0(S(I_i),S(I_j))} {\sqrt{I_i}\sqrt{I_j}} = \frac{I_i}{\sqrt{I_iI_j}} = \sqrt{\frac{I_i}{I_j}}.

The same model gives a useful one-step conditional distribution. Since the pair $(Z_{j-1},Z_j)$ is jointly normal with correlation $\sqrt{I_{j-1}/I_j}$,

Z_j\mid Z_{j-1}=z \sim N\left( \sqrt{\frac{I_{j-1}}{I_j}}\,z,\; 1-\frac{I_{j-1}}{I_j} \right).

The second argument of $N$ is the variance. Unequal information increments therefore change both the conditional mean coefficient and the conditional variance from one interval to the next. The stagewise tail can be evaluated using the multivariate normal probabilities or by conditioning successively with these transitions. Either way, the actual $I_j$ values and all relevant boundaries must be used.

Worked Example: A Two-Look Trial With Unequal Information

Suppose a hypothetical trial has information fractions $t_1=I_1/I_2=0.35$ and $t_2=1$, with upper boundaries $b_1=2.50$ and $b_2=2.00$. The trial does not cross at look 1 and stops for efficacy at look 2 with $z_2=2.20$. The canonical model is assumed, the information is increasing, and the observed path satisfies $Z_1<2.50$, so it is eligible to reach look 2.

The null correlation is $r=\sqrt{0.35}=0.5916$. Under the stagewise ordering, the look-1 stop probability is $P(Z_1\geq2.50)=0.00621$. The look-2 contribution is the probability of reaching look 2 and attaining $Z_2\geq2.20$:

P(Z_1<2.50,Z_2\geq2.20) = P(Z_2\geq2.20)-P(Z_1\geq2.50,Z_2\geq2.20).

Both marginal statistics are standard normal, so $P(Z_2\geq2.20)=0.01390$. Evaluating the bivariate standard normal tail with correlation 0.5916 gives $P(Z_1\geq2.50,Z_2\geq2.20)\approx0.00134$. Thus the look-2 contribution is $0.01390-0.00134=0.01256$, and

p_{\mathrm{stage}} \approx 0.00621+0.01256=0.01877.

The stagewise p-value is approximately 0.0188. It is not the single-look tail $P(Z_2\geq2.20)=0.0139$: it also includes the probability of an earlier efficacy stop, as required by this ordering. All probabilities here are null probabilities for the stated hypothetical schedule and stopping rule.

Calculations Across More Than Two Looks

At a later stopping look, each earlier stop probability is a probability over a region of the joint distribution. For three looks, for example, the first two stop probabilities are $P(Z_1\geq b_1)$ and $P(Z_1<b_1,Z_2\geq b_2)$. The final contribution for a look-3 observation $z_3$ is $P(Z_1<b_1,Z_2<b_2,Z_3\geq z_3)$. The non-crossing constraints cannot be omitted: the observed path did not stop at either earlier look.

Worked Example: An Early Stop at Look Two

Consider a hypothetical three-look design with information fractions $0.20,0.65,1.00$, and boundaries $b_1=2.80$, $b_2=2.30$, and $b_3=2.00$. Suppose the observed trial has $Z_1<2.80$ and stops at look 2 with $z_2=2.45$. The look-3 information and boundary do not enter the ordered tail for this observed look-2 outcome.

The look-1 null probability is $P(Z_1\geq2.80)=0.00256$. The correlation between the first two statistics is $r_{12}=\sqrt{0.20/0.65}=0.5547$. The look-2 contribution is $P(Z_1<2.80,Z_2\geq2.45)$. Since $P(Z_2\geq2.45)=0.00714$, and the bivariate upper-tail probability $P(Z_1\geq2.80,Z_2\geq2.45)$ is approximately 0.00035, the contribution is $0.00714-0.00035=0.00679$.

Therefore, $p_{\mathrm{stage}}\approx0.00256+0.00679=0.00935$. The first term counts all earlier efficacy stops; the subtraction in the second term removes paths that would already have stopped at look 1. The correlation used here is determined by the information ratio, not by the number of looks.

Worked Example: A Look-Three Stop With Unequal Increments

For another hypothetical design, let the information fractions be $0.25,0.60,1.00$, with boundaries $b_1=2.60$, $b_2=2.40$, and $b_3=2.10$. The observed path satisfies $Z_1<2.60$, $Z_2<2.40$, and stops at look 3 with $z_3=2.30$. The information ratios imply correlations $r_{12}=\sqrt{0.25/0.60}=0.6455$, $r_{13}=0.5000$, and $r_{23}=\sqrt{0.60}=0.7746$.

The stagewise p-value is the sum of the first-look stop probability, the second-look stop probability, and the probability of reaching look 3 with $Z_3\geq2.30$:

p_{\mathrm{stage}} = P(Z_1\geq2.60) + P(Z_1<2.60,Z_2\geq2.40) + P(Z_1<2.60,Z_2<2.40,Z_3\geq2.30).

For the specified canonical correlation matrix, multivariate normal probability evaluation gives, to the displayed precision, 0.00466 for the first term, 0.0073 for the second, and 0.0062 for the third. Hence $p_{\mathrm{stage}}\approx0.00466+0.0073+0.0062=0.0182$. These three terms are disjoint stopping-stage contributions under the null. The third term is not the marginal tail $P(Z_3\geq2.30)$, because it excludes paths that crossed either earlier boundary.

Checking the Calculation and Choosing an Approach

The canonical joint-normal calculation is appropriate only when the assumed information levels, test-statistic construction, and stopping rule describe the analysis. Before reporting a result, check that information fractions are strictly increasing, the covariance matrix uses $\sqrt{I_i/I_j}$ for $i<j$, and the observed path satisfies every earlier non-crossing condition. The p-value must also correspond to the stated ordering. Asymmetrical boundaries can be handled by using the actual $b_j$ values in the same probability regions; the correlation formula does not change.

An equal-information design has $I_j=jI_1$, and therefore a particular correlation structure follows from that schedule. Replacing an unequal schedule by equal spacing changes the joint null distribution and generally changes the stopping-region probabilities. A single-look normal tail is simpler, but it does not account for the ordered stopped outcomes. Conversely, likelihood-ratio ordering is a different way to order stopped outcomes and can give a different sequential p-value; it should not be substituted silently for the stagewise ordering used here.

A useful diagnostic is to compare each computed contribution with its event definition. Every stop probability must be nonnegative, and the disjoint stop probabilities through look $k$ must sum to at most one. The final contribution must be no greater than the marginal tail $P(Z_k\geq z_k)$, since it adds non-crossing restrictions. A value exceeding that marginal tail indicates a region, correlation, or arithmetic error. For an observed efficacy stop, the result should include the earlier stop probabilities under the chosen ordering; omitting them is another warning sign.

Common mistake: Treating unequal looks as equally spaced because there are the same number of interim analyses, then using an equal-spacing correlation matrix. The look count does not determine dependence: the information ratios do. Use $\operatorname{Corr}_0(Z_i,Z_j)=\sqrt{I_i/I_j}$, and calculate probabilities over the actual stopping regions. A stagewise p-value is specific to the assumed null model, schedule, boundaries, and ordering.

The method has limits. It does not by itself resolve uncertainty in the information levels, accommodate a misspecified statistic, or define a unique p-value independent of the ordering. If the actual look schedule or stopping rules differ from those used in the calculation, the resulting probabilities may not describe the observed design. The calculation also depends on a multivariate normal model (exact in the canonical Gaussian setting and approximate in many other settings); model adequacy must be considered in context.

Check Your Understanding

Answer each question using the canonical model and the stagewise ordering defined above. Brief solutions follow.

If $I_1/I_2=0.49$, what is $\operatorname{Corr}_0(Z_1,Z_2)$? Solution: It is $\sqrt{0.49}=0.7000$.
For an observed look-2 stop, why is $P(Z_2\geq z_2)$ not generally the complete stagewise p-value? Solution: It includes paths that crossed the look-1 boundary and would not have reached look 2; the stagewise p-value also includes earlier stops according to the ordering.
With $I_1/I_2=0.25$, state the conditional distribution of $Z_2$ given $Z_1=z$. Solution: The conditional mean is $\sqrt{0.25}z=0.5z$, and the conditional variance is $1-0.25=0.75$. Thus $Z_2\mid Z_1=z\sim N(0.5z,0.75)$.
For a look-3 stop, what event defines the final contribution when earlier boundaries are $b_1,b_2$ and the observed statistic is $z_3$? Solution: It is $P_0(Z_1<b_1,Z_2<b_2,Z_3\geq z_3)$; both earlier non-crossing conditions are required.
A calculation gives a final contribution larger than $P(Z_k\geq z_k)$. What should be checked? Solution: The constrained event is a subset of the marginal-tail event, so its probability cannot be larger. Check the covariance matrix, inequalities defining the event, and numerical evaluation.