Why Repeated Looks Change the Stagewise Evidence
A stage in a clinical trial may contain several interim analyses. Investigators might examine accumulating outcomes at two or more scheduled looks and stop early if the evidence is sufficiently strong. The result entering a multi-stage combination test must account for the opportunity to cross a boundary at any of those looks. A p-value calculated as if the most favorable look had been chosen in advance generally does not do so.
The goal is to summarize the entire within-stage path with a p-value that remains valid under the stagewise null hypothesis. Validity means that, for every \(a\) between 0 and 1, the probability under the null of obtaining a p-value no greater than \(a\) is at most \(a\). Such a p-value can then serve as a stage input to a combination test, subject to the combination test's assumptions about the stage inputs.
Suppose a stage has \(m\) planned looks, and let the observed path include the test statistics or p-values at all looks actually reached. For each nominal level \(a\), define a rejection region \(R_a\) for that complete path. The regions should be nested: if \(a_1\leq a_2\), then \(R_{a_1}\subseteq R_{a_2}\). They must also be calibrated under the stagewise null, with \(P_0(\text{observed path}\in R_a)\leq a\) for every \(a\in[0,1]\). A path-based stagewise p-value is the infimum of the levels at which the path enters these regions:
The infimum is used rather than a smallest level: a path can enter every region for levels strictly above a boundary without belonging to the region at the boundary itself. Under the null calibration and nestedness, for \(a<1\), \(\{P_s\leq a\}\subseteq\bigcap_{n:\,a+1/n\leq1}\{\text{observed path}\in R_{a+1/n}\}\), so continuity from above gives \(P_0(P_s\leq a)\leq a\); the boundary convention must preserve this validity. Continuous null distributions often make equality at a boundary probability zero; with discrete data, the calibration must handle boundary mass so that \(P(P_s\leq a)\leq a\) still holds.
Constructing a Stagewise P-Value With Alpha Allocation
A transparent approach is to divide the stage's nominal error level among its looks. Let \(p_i\) be a valid one-sided p-value at look \(i\), calculated as though that look were the only planned test. Choose fixed nonnegative allocation weights \(w_1,\ldots,w_m\) with \(\sum_{i=1}^m w_i=1\). At stagewise level \(a\), reject if at least one look satisfies \(p_i\leq w_i a\). The corresponding stagewise p-value is
where a look assigned weight zero is not allowed to trigger rejection. This construction does not require independence between looks. It does require that the look-specific p-values are valid under the null and that the allocation weights and look plan are fixed independently of the observed results.
Worked Example: Unequal Error Allocation Across Two Looks
In a hypothetical trial stage, a prespecified one-sided test is conducted at two interim looks. The allocation weights are \(w_1=0.25\) and \(w_2=0.75\). The observed standardized statistics are \(Z_1=1.80\) and \(Z_2=2.20\). Assume that, under the null, each statistic has a standard normal marginal distribution, so its one-sided p-value is \(1-\Phi(Z_i)\).
The look-specific p-values are \(p_1=1-\Phi(1.80)\approx0.03593\) and \(p_2=1-\Phi(2.20)\approx0.01390\). Therefore,
At stagewise level \(a=0.02\), the look-specific thresholds are \(w_1a=0.005\) and \(w_2a=0.015\). The first look does not cross its threshold, but the second does because \(0.01390\leq0.015\). The path therefore rejects at level 0.02, consistent with its stagewise p-value of about 0.0185. This calculation is valid even if the two looks are correlated.
Why the Allocation Construction Controls Error
The justification is the union bound, which states that the probability of a union of events is at most the sum of their probabilities. At a fixed level \(a\), rejection occurs if any event \(A_i=\{p_i\leq w_i a\}\) occurs. Since each \(p_i\) is valid, \(P(A_i)\leq w_i a\). Thus,
This proves stagewise type I error control without assuming independence across looks. The guarantee may be conservative because the events can overlap. More efficient procedures can use the joint null distribution of the repeated-look statistics to calibrate the probability of crossing at least one boundary, but that calibration must reflect the dependence induced by accumulating data.
Worked Example: Three Looks and the Selection Effect
Consider a hypothetical stage with three prespecified looks, equal allocation weights \(w_i=1/3\), and observed one-sided statistics \(Z_1=2.45\), \(Z_2=2.10\), and \(Z_3=1.60\). Assume standard normal marginal null distributions. Their one-sided p-values are approximately \(0.00714\), \(0.01786\), and \(0.05480\).
The smallest look-specific p-value is \(0.00714\), but reporting that as the stagewise p-value would ignore the chance of obtaining a favorable result at any of three looks. The allocated stagewise p-value is
Thus the stage rejects at level \(0.025\), but not at level \(0.02\). The unadjusted value \(0.00714\) would incorrectly imply rejection at the much smaller level 0.01. The union-bound adjustment is valid regardless of the dependence among the three looks, though it may be more conservative than an exact joint calibration.
Using the Stagewise P-Value in a Combination Test
Once each stage has a valid p-value, a prespecified combination rule can combine the stage inputs. For example, an inverse normal combination test transforms each one-sided p-value \(P_s\) to \(Z_s=\Phi^{-1}(1-P_s)\), then forms a weighted sum. With two stages and fixed weights \(v_1,v_2\) satisfying \(v_1^2+v_2^2=1\), the statistic is \(C=v_1Z_1+v_2Z_2\). Under independent, continuous, exact uniform stagewise p-values under the global null, \(C\) has a standard normal distribution. Under valid but conservative p-values and nonnegative combination weights, the test is generally conservative under the corresponding independence assumptions.
A repeated-look stagewise p-value replaces the stage's look-by-look results; the combination test should not treat every interim look as a separate stage input unless its calibration was designed for that structure. The validity of the within-stage summary and the validity of the multi-stage combination are distinct requirements.
Worked Example: Combining a Repeated-Look Stage With a Second Stage
In a hypothetical two-stage trial, stage 1 has two planned looks with equal error-allocation weights. Its observed one-sided statistics are \(Z_{11}=2.45\) and \(Z_{12}=2.10\), giving look-specific p-values approximately \(0.00714\) and \(0.01786\). Hence its valid Bonferroni stagewise p-value is \(P_1=2(0.00714)=0.01428\), rounded to \(0.0143\). Suppose the independent second-stage data yield a valid one-sided stagewise p-value \(P_2=0.200\). Use fixed combination weights \(v_1=0.600\) and \(v_2=0.800\); their squared sum is \(0.600^2+0.800^2=1\).
The normal scores are
The combined statistic and one-sided p-value are
At a 0.05 significance level, the combined test rejects the global null. This conclusion relies on the stated independence of the stage data under the null, valid stagewise p-values, and weights fixed independently of the observed results. The p-value of about 0.0235 is evidence against the global null under this design; it is not the probability that the null hypothesis is true.
Calibration Choices, Diagnostics, and Limits
A direct alternative to the union bound is to use the joint null distribution of the full repeated-look path. For example, one may choose a statistic such as the maximum of the standardized statistics and calculate its null tail probability using their joint distribution. This can improve power by accounting for correlations between looks, but requires a correctly specified joint model and a calibration that includes the stopping rule. Alpha allocation is simpler and robust to look dependence, at the cost of possible conservatism.
Before using a stagewise p-value, verify that the look schedule, one-sided direction, allocation weights or joint calibration, and stopping boundaries were specified appropriately. Check that the analysis includes all looks actually examined, including any unplanned analyses that influenced decisions. If investigators inspect results more often than the stated schedule, calibrating only for the scheduled looks does not account for that additional opportunity to stop.
The usual combination-test result also needs suitable stagewise inputs. In particular, the stages should provide independent evidence under the null when the chosen reference distribution assumes independence. Reusing participants or outcomes across stages can induce dependence and invalidate a reference distribution that assumes independent inputs. An individual stagewise p-value can be valid while the combination calibration is not.
Alpha allocation is not the only valid construction, and a valid p-value need not be exactly uniform. Discrete outcomes, conservative look-level tests, or conservative multiple-look bounds can produce p-values with \(P(P_s\leq a)<a\) at some levels. This is acceptable for type I error control, though it can reduce power. A different calibrated procedure may be preferable when a justified joint null model is available.
Common Mistake
Wrong reasoning: “The smallest p-value across the interim looks is 0.0071, so the stagewise evidence is 0.0071.” This treats the most favorable look as if it had been selected in advance and ignores repeated opportunities to cross a threshold.
Correction: Define the rejection rule for the complete within-stage path, then calculate a p-value calibrated for that rule. With \(m\) equally allocated looks, the Bonferroni construction uses \(\min(1,m\min_i p_i)\). If a joint boundary-crossing probability is used instead, it must account for the dependence and stopping rule across all looks.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Three looks have valid one-sided p-values 0.03, 0.08, and 0.12. With equal allocation weights, what is the Bonferroni stagewise p-value, and would the stage reject at level 0.05?
- Why does the alpha-allocation construction not require independence between looks?
- What error is made by reporting only the smallest look-specific p-value after examining all planned looks?
- Name two conditions needed for the standard independent-stage inverse normal reference distribution after within-stage looks have been summarized.
Solutions. (1) The stagewise p-value is \(\min(1,3(0.03))=0.09\), so the stage does not reject at 0.05. (2) The union bound controls the probability of crossing any look's threshold using the sum of the marginal probabilities; it does not require the events to be independent. (3) That value ignores the repeated opportunity to obtain a small p-value and can understate the stagewise type I error. (4) The stagewise p-values must be valid under the null, and the stage evidence must be independent across stages under the null when the reference distribution assumes independence; the combination weights must also be fixed independently of observed results.