Why the Distinction Matters
In a multi-endpoint group-sequential design, a result can look “significant” against a familiar nominal threshold without satisfying the trial’s prespecified rule for rejecting that endpoint’s null hypothesis. This distinction matters whenever rejection at one endpoint is a condition for reallocating alpha to another endpoint. A small nominal p-value is evidence against a null hypothesis at one look; it is not, by itself, permission to change the testing procedure.
The relevant decision is whether the endpoint’s sequential test rejects at the current look. That decision depends on the prespecified boundary, the look, and—where applicable—the path through earlier looks. Alpha recycling must use that rejection event as its gate. Replacing it with “the p-value was below .05” or another nominal threshold substitutes a different decision rule.
This tutorial uses upper-tail statistics for clarity. For a lower-tail or two-sided test, the same principle applies using the corresponding prespecified rejection region. A nominal threshold is meaningful only when its direction and tail convention are specified.
What the Sequential Rule Requires
A group-sequential procedure plans analyses at specified looks and uses boundaries calibrated for those repeated opportunities to examine the data. For endpoint \(j\), the rejection decision at look \(k\) is governed by the boundary \(b_{jk}\), not automatically by the fixed-look critical value for a familiar nominal level. The boundary may be stricter or less strict than that reference value, depending on the design and the look.
A multiple-endpoint procedure adds another layer. In a graph-based procedure, for example, an endpoint’s rejection may trigger a prespecified transfer of testing weight or alpha to other hypotheses. The transfer rule names the event that permits the transition: the endpoint’s null is rejected by its allocated, sequentially valid test. It does not say that any small-looking p-value opens the gate. The rule must also use the planned boundary and its specified treatment of earlier looks.
The design’s error-control claims rely on the full decision rule, including its rejection boundaries, the way tests are combined across looks and endpoints, and the conditions for any alpha transfers. Valid marginal p-values alone do not establish that an improvised recycling rule controls the familywise error rate. The joint behavior of the test statistics and the prespecified procedure matter, as emphasized in the earlier material on dependence assumptions in intersection tests and endpoint-specific test statistics and local boundaries.
Before applying the rule, confirm that the data and analysis match the design: the endpoint and direction of testing are specified; the look is planned; the statistic is calculated as prespecified; and the boundary and any transfer weights are the calibrated ones. If an endpoint is unavailable or its analysis is not yet permitted, a nominal p-value cannot make it eligible for a transfer.
A Short Justification: Why Nominal Significance Is Not the Gate
Suppose the test is one-sided and \(Z_{jk}\) has a standard normal distribution under the null hypothesis at a given look. A fixed-look nominal threshold \(c\) rejects when the nominal p-value is at most \(c\), equivalently when \(Z_{jk}\ge z_{1-c}\), where \(z_{1-c}\) is the \(1-c\) quantile of the standard normal distribution. The sequential test at that look rejects when \(Z_{jk}\ge b_{jk}\), subject also to the design’s stopping history.
When \(b_{jk}>z_{1-c}\), a statistic can fall between those two cutoffs. In that case it is nominally significant but does not cross the sequential boundary. Ignoring stopping history can create another mismatch: a statistic at a later look must be interpreted according to the rule for the path actually taken, not as though the trial had planned only that one analysis.
Thus, in this common case, the nominal-significance event is broader than the rejection event. Using it as the condition for recycling permits transitions on paths where the endpoint was not rejected by the design. This does not prove that every such substitution will inflate the familywise error rate by a particular amount. It does show that the substituted procedure is not the prespecified procedure, so its error-control justification no longer automatically applies.
Worked Example: Nominally Significant, but Below the Interim Boundary
Suppose a hypothetical trial uses a one-sided \(Z\)-statistic for endpoint A at an interim look. The prespecified boundary is \(b_{A1}=2.80\). A nominal one-sided threshold of \(0.025\) corresponds to a standard normal cutoff of about \(1.960\). The observed statistic is \(Z_{A1}=2.10\). Does A reject, and can its rejection trigger a transfer?
The nominal p-value is \(1-\Phi(2.10)\approx 0.0179\). Since \(0.0179<0.025\), the result is nominally significant at the stated reference threshold.
The sequential decision uses the interim boundary: \(2.10<2.80\). Therefore A does not reject at this look. If the prespecified graph permits a transfer only after A’s sequential rejection, no transfer is permitted. The nominal p-value does not override the boundary.
This calculation assumes the one-sided statistic and direction were specified, the interim look is the planned look, and the boundary of 2.80 is the applicable calibrated boundary. If any of those conditions is false, the stated decision rule cannot simply be applied to these numbers.
Follow the Gate, Not the Familiar Threshold
A useful way to implement the rule is to treat the sequential rejection as a logical gate. First decide whether the endpoint rejects under its own prespecified test. Only then apply the graph’s stated transition. Do not change the gate because the nominal p-value is close to, or below, a conventional value.
Confirm the endpoint, direction, look, statistic, and boundary specified for this analysis.
Apply the prespecified boundary and stopping history to determine whether the endpoint rejects at this look.
If the rule requires rejection, transfer alpha only when the sequential decision is rejection. Otherwise keep the testing state specified for a nonrejection.
Report the nominal p-value if useful, but do not let it replace the rejection indicator in the transfer rule.
Worked Example: A Nominal Result Does Not Open the Graph Gate
Consider a hypothetical two-endpoint design with an interim rule specified in advance. Endpoint A rejects at this look only if \(Z_A\ge2.58\). If A rejects, endpoint B is tested at a one-sided cutoff of \(1.960\); if A does not reject, B uses its prespecified cutoff of \(2.576\). Assume these rules and cutoffs are part of a procedure calibrated in advance. The observed statistics are \(Z_A=2.40\) and \(Z_B=2.40\).
For A, \(1-\Phi(2.40)\approx0.0082\), which is below the nominal reference threshold \(0.025\). But \(2.40<2.58\), so A does not reject under the stated sequential rule. The graph gate remains closed, and B must use the no-transfer cutoff of \(2.576\).
For B, \(2.40<2.576\), so B does not reject under that cutoff. If nominal significance were incorrectly substituted for A’s rejection, B would instead be tested at \(1.960\); then \(2.40\ge1.960\) and B would reject. That is a different decision path from the one specified. The fact that B would pass the relaxed cutoff does not authorize using it: the required rejection by A did not occur.
This example illustrates the logic of a gate, not a general recipe for choosing cutoffs. The cutoffs and transfer rule must be justified together under the actual design’s assumptions about the statistics and their dependence.
Worked Example: Recycling Can Become Available at a Later Look
Suppose a hypothetical endpoint has two planned looks and upper boundaries \(b_{A1}=2.80\) and \(b_{A2}=2.10\). Its prespecified rule stops for rejection at the first boundary crossed. At look 1, \(Z_{A1}=2.35\); at look 2, if the trial continues, \(Z_{A2}=2.15\). For illustration, compare each statistic’s nominal one-sided p-value with \(0.025\).
At look 1, \(1-\Phi(2.35)\approx0.0094\), so the result is nominally below \(0.025\). However, \(2.35<2.80\); there is no sequential rejection at look 1 and therefore no transfer at that look.
Because there was no earlier rejection, the trial continues to look 2 under this rule. There, \(1-\Phi(2.15)\approx0.0158\), again below \(0.025\), and \(2.15\ge2.10\). The sequential rule rejects at look 2, so a transfer is permitted at that point if the prespecified multiplicity procedure says this rejection triggers one.
The conclusion depends on the stated boundaries and continuation rule. The nominal p-values alone would not tell us when the transfer becomes available. In an actual design, the boundaries must be calibrated for its planned looks, information, and multiplicity procedure; this pair of illustrative boundaries is not a recommendation for use.
Diagnostics, Alternatives, and Limitations
A practical diagnostic is to compare the recorded transfer indicator with the endpoint’s formal rejection indicator at every look. If the transfer indicator turns on when the sequential test did not reject, the implementation has used the wrong gate. A second check is to verify the boundary version, look number, testing direction, and stopping history used in the calculation. A p-value calculated for the wrong tail or look can make even a correctly specified rule appear inconsistent.
The main alternative to boundary-based decisions is to report a sequentially adjusted p-value, when one is available for the procedure. Such a value is constructed to reflect the planned sequential testing rule; compare it with its corresponding prespecified criterion, rather than treating an unadjusted nominal p-value as interchangeable. A design with only one prespecified analysis may instead use a fixed-look test. That is a different analysis plan, not a justification for replacing a group-sequential boundary after seeing interim data.
Nominal p-values can still be useful descriptive summaries. They communicate how extreme the statistic is relative to a single-look reference distribution. The limitation is not that nominal p-values are inherently wrong; it is that they answer a different question from whether the endpoint has rejected under its sequential, multiple-endpoint procedure. Neither a nominal p-value nor crossing a boundary should be interpreted as the probability that a null hypothesis is true.
The precise effect of an incorrect gate depends on the full procedure, including the joint distribution of endpoint statistics and the transfer rules. A nominal-significance gate may create additional downstream testing paths. Whether those paths violate a particular error-rate guarantee must be assessed for the whole procedure; one cannot infer a numerical error rate from a single p-value. The safe implementation principle is exact: apply only the rejection and transfer conditions specified and validated for the design.
Check Your Understanding
Answer each question using the stated rule, then compare with the solutions.
- A one-sided interim boundary is 2.70. The observed statistic is 2.20, with nominal p-value 0.0139. Does the endpoint reject? If rejection is required for recycling, is alpha transferred?
- For a one-sided test, the nominal threshold is 0.025, corresponding to a cutoff of approximately 1.960. The sequential boundary is 2.40. Give an interval of statistics that are nominally significant but do not cross the boundary.
- An endpoint did not reject at an earlier look. At the current planned look, its statistic crosses the applicable boundary. When may a prespecified transfer occur?
- Why is replacing the sequential rejection condition with a nominal p-value condition not automatically justified by the validity of the endpoint’s marginal p-value?
Solutions. 1. No. Although \(0.0139<0.025\), \(2.20<2.70\), so there is no sequential rejection and no transfer under a rule that requires rejection. 2. Any statistic \(Z\) with \(1.960\le Z<2.40\) is nominally significant at 0.025 but below the sequential boundary. 3. The endpoint may reject at the current look if the planned stopping rule and boundary are satisfied; a transfer may then occur only if the prespecified multiplicity rule says that this rejection triggers it. 4. A marginal p-value describes evidence at a single look, whereas the sequential decision incorporates the boundary and planned looks. Substituting the broader nominal event can add paths not included in the validated rule, so the error-control justification for the original procedure need not apply.