Why a Graph Transition Requires a Boundary Check
A multiplicity graph can change the local alpha available to an endpoint after a hypothesis is rejected. In a group sequential design, however, that endpoint may already have been tested at earlier looks. A new allocation is not automatically a fresh error budget: the procedure must account for previous opportunities to reject, for the endpoint’s current rejection status, and for the information time at which the next analysis occurs.
The audit question is therefore not simply, “What is the endpoint’s alpha now?” It is, “Does the complete rejection rule, including the new allocation and the actual analysis schedule, still control the probability of a false rejection?” The introductory material on spending alpha after a graph transition establishes the allocation context; here the emphasis is on checking sequential boundaries after the transition.
Let \(Z_{jk}\) denote the test statistic for endpoint \(j\) at look \(k\), and let \(c_{jk}\) be its specified boundary. For a one-sided test in which large values favor \(H_a\), the local rejection rule is \(Z_{jk} \ge c_{jk}\). A two-sided rule must specify both boundaries or an equivalent two-sided p-value rule. The statistics and boundaries must use the endpoint’s stated direction, variance model, and analysis timing.
Write \(R_{jk}\) for the event that \(H_j\) has been rejected by look \(k\). Since rejection at an earlier look makes the same hypothesis ineligible for another rejection, the later testing event is restricted to paths on which \(R_{j,k-1}\) did not occur. A valid sequential procedure accounts for the probability of crossing at any eligible look, not just the probability associated with its final boundary.
A Validity Checklist
A practical check separates the multiplicity allocation from the sequential calibration. First verify that the graph transition is authorized by the prespecified procedure. Then verify that each affected endpoint’s updated local test controls its error over all of its remaining eligible looks. The graph’s familywise guarantee depends on both parts; a correctly updated graph allocation cannot repair a miscalibrated sequential test.
Identify which rejection triggered the transition, which endpoints remain eligible, and the local allocations authorized by the graph rule. Do not treat an endpoint that has already rejected its null as eligible for later testing.
For each still-eligible endpoint, identify the probability of a false crossing through the current look under the actual sequential rule. Do not substitute an observed p-value for this probability.
Confirm that the updated boundaries or conditional rejection thresholds use only future eligible looks and leave no more error than the authorized remaining budget.
Use the actual information time for each endpoint and the correct joint or conditional null distribution. A boundary calculated for a different look schedule is not automatically valid.
A particularly transparent way to audit a local rule uses conditional p-values. Let \(\mathcal F_{k-1}\) denote all information available before look \(k\), including the history that determined whether the endpoint is eligible and what threshold applies. Suppose \(P_{jk}\) is a conditionally valid p-value under \(H_j\), meaning \(P(P_{jk}\le u\mid\mathcal F_{k-1})\le u\) for every \(u\) between 0 and 1. If the procedure tests at look \(k\) only when eligible, with a nonnegative threshold \(b_{jk}\) determined before observing that look’s test result, then
Here \(I(\cdot)\) equals 1 when the condition holds and 0 otherwise. Summing over the looks bounds the probability of at least one false rejection by the expected total threshold used on eligible paths. In particular, if the sum of the thresholds used on every path is at most the authorized local cap \(\alpha_j\), the endpoint’s false-rejection probability is at most \(\alpha_j\). This is a conservative but useful justification: it makes clear why a transition must affect only the future budget, and why the threshold must be set before the corresponding result is seen.
Conventional group sequential boundaries are often calibrated by a spending function under a model for the joint distribution of the statistics across looks, rather than by applying this simple conditional-p-value construction. In that case, the audit must use that same model and calibration. The probability of crossing by look \(k\) is the probability of the union of eligible crossing events under the null; it is not generally obtained by adding the nominal p-value cutoffs.
Worked Example: A Transition for an Endpoint That Has Not Rejected
Consider a hypothetical two-endpoint design with familywise level \(0.05\). Before an interim analysis, the graph assigns local caps \(0.03\) to \(H_A\) and \(0.02\) to \(H_B\). The planned local sequential procedure for \(H_B\) has used \(0.004\) of its error allowance by the interim look. At that look, the one-sided p-value for \(H_A\) is \(0.004\), below its prespecified interim boundary of \(0.006\), so \(H_A\) is rejected. The p-value for \(H_B\) is \(0.012\), above its interim boundary of \(0.010\), so \(H_B\) is not rejected and remains eligible.
Suppose the prespecified graph rule transfers \(0.010\) of local alpha to \(H_B\) after rejection of \(H_A\). Its updated local cap is \(0.020+0.010=0.030\). Since \(H_B\) has already used \(0.004\), the arithmetic remainder under this accounting is
That \(0.026\) is a ceiling on the future error allowance under the stated accounting, not an automatic p-value cutoff for the next look. The future boundary must be calibrated for the remaining look schedule and the distribution of \(H_B\)’s statistics, conditional on the relevant history or by an approved sequential spending method. Because \(H_B\) did not reject at the interim, a future test is meaningful; if it had rejected, it would no longer be eligible for testing.
For this illustration, assume the graph rule is valid for the endpoint dependence in the design, and that the future local calibration accounts for the history that determines eligibility and transfer. Those assumptions matter: if the endpoints are dependent, a conditional p-value for \(H_B\) must be valid given the information used to trigger the transition, or the joint calibration must otherwise justify the rule. Simply subtracting the interim expenditure and declaring the next nominal p-value threshold to be \(0.026\) would not establish that validity.
Analysis Timing Is Part of the Boundary
Sequential boundaries depend on when information is observed. Let \(t_k\) be the information fraction at look \(k\), with \(0<t_1<\cdots<t_K=1\). A spending function \(A_j(\alpha_j,t)\) specifies the planned cumulative type I error spending by information time \(t\) for local level \(\alpha_j\). Under a conventional group sequential calibration, the critical boundary at each look is chosen so that the probability of having crossed by that look matches the spending target, under the assumed null distribution of the sequential statistics.
A transition that changes \(\alpha_j\) can therefore require recalibration of future boundaries. So can an endpoint analysis that occurs at a different information fraction from the common schedule. The calendar date alone is not enough: one endpoint may have accrued more information than another. Reusing the old boundary without checking its information-time calibration can either spend too much or leave some of the authorized error unused.
Worked Example: Checking a Shifted Information Time
Suppose a hypothetical endpoint has local level \(\alpha_j=0.020\) and uses the illustrative linear spending target \(A_j(\alpha_j,t)=\alpha_j t\). Its planned interim information fraction is \(t=0.50\), so the target cumulative spending is
The endpoint’s analysis is delayed until its information fraction is \(t=0.60\). At that actual time, the target is
The difference in cumulative targets is \(0.012-0.010=0.002\). This calculation identifies the amount by which the planned spending target changes; it does not by itself give a valid critical z-value or p-value boundary. That boundary must be recalibrated for the actual information time and the assumed distribution of the endpoint statistic. The linear function is illustrative, not a claim that all designs should use linear spending.
For this check, assume the information fraction is estimated appropriately, the endpoint’s statistic has the null distribution used for calibration, and the look is conducted according to the stated testing rule. If the observed information fraction is unknown or the analysis is performed at a different time, the design needs a rule for handling that deviation; silently looking up the planned \(t=0.50\) boundary is not a calibration check.
Distinguishing a Boundary From a Nominal Cutoff
A sequential boundary can be reported as a critical value for a statistic, a nominal p-value at a look, or a conditional error threshold. These quantities are related but not interchangeable. In particular, a look-specific nominal p-value cutoff is not generally the same as the probability spent at that look. Spending functions describe cumulative crossing probabilities under a null model. The boundaries are calculated to achieve those probabilities while accounting for dependence between looks.
Worked Example: Auditing Two Conditional Spending Increments
Consider a hypothetical endpoint with a local cap of \(0.015\), tested at two looks using valid conditional p-values. The prespecified thresholds are \(0.004\) at look 1 and \(0.011\) at look 2, so their sum is \(0.015\). At look 1, the conditional p-value is \(0.005\), which exceeds \(0.004\); the endpoint is not rejected. At look 2, suppose its conditional p-value is \(0.010\), which is at most \(0.011\); the endpoint is rejected at look 2.
Under the stated assumption that each conditional p-value is super-uniform under the null given the full prior history, the false-rejection probability at look 1 is at most \(0.004\), and the probability of a false rejection at look 2 is at most \(0.011\) on paths eligible for that look. Thus the union bound gives
This is a valid conservative spending argument for the specified conditional-p-value rule. It does not imply that ordinary group sequential nominal p-value cutoffs can always be added this way. If the thresholds were instead boundaries calibrated from a joint distribution of correlated z-statistics, the crossing probability should be checked using that calibration, not inferred by summing the nominal p-values.
Diagnostics, Alternatives, and Limits
A boundary audit should flag the following patterns before a result is reported:
- A stale boundary: the local allocation changed, but the future boundary is still the one calibrated for the old level.
- A fresh-budget mistake: the new allocation is used in full even though the endpoint already had earlier testing opportunities.
- An ineligible endpoint: a future rejection rule is applied to a hypothesis already rejected by the procedure.
- A timing mismatch: the boundary is taken from the planned look rather than the endpoint’s actual information time.
- A nominal-p-value shortcut: p-value cutoffs are added or compared with a local cap as if they were cumulative crossing probabilities.
- An unaccounted dependence structure: the calibration ignores correlations across looks or the information that triggered a graph transition.
There are several ways to verify a rule. A local spending-function calculation is natural when the endpoint’s look schedule and null distribution are specified and the spending function is fixed. Conditional-error recalibration can be useful after a design change: it preserves a bound on the probability of future rejection conditional on the observed history, provided that conditional distribution is correctly specified. Joint simulation can check a complex graph and sequential procedure under specified endpoint dependence and timing scenarios. Simulation is a diagnostic, not a substitute for a validity argument: finite simulations cannot prove that the error rate is bounded, and the scenarios must include relevant null configurations.
These approaches answer related but different questions. A spending-function calculation checks planned cumulative error over looks. Conditional-error methods focus on the remaining rejection opportunity given observed history. Joint simulation estimates the behavior of the complete procedure, including its transitions. Whichever approach is used, the rejection rule, information times, dependence assumptions, and treatment of endpoints already rejected must be explicit. A boundary check alone does not establish that the multiplicity graph controls familywise error; the graph’s allocation and transition rules must also be valid.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An endpoint’s local cap rises from \(0.018\) to \(0.025\) after a graph transition. It has already spent \(0.005\). What is the arithmetic remainder, and why is it not automatically the next p-value cutoff?
- At an interim look, an endpoint’s p-value is below its boundary and the endpoint is rejected. May the same procedure test that hypothesis again at the final look?
- A boundary was calibrated at information fraction \(0.40\), but the endpoint is analyzed at \(0.55\). What should be checked before using it?
- Why can nominal p-value cutoffs from a conventional group sequential design not generally be added to determine cumulative type I error spending?
Solutions. (1) The remainder is \(0.025-0.005=0.020\). It is an upper allowance for a future calibrated rule, not automatically a single-look cutoff. (2) No. Once rejected, the hypothesis is no longer eligible for rejection under that procedure. (3) Check the actual information-time spending target and recalibrate the boundary under the design’s assumed null distribution and analysis rule; the planned-time boundary is not automatically valid. (4) The cutoffs are not generally probabilities of disjoint crossing events. Cumulative error depends on the joint distribution of statistics across looks and must be obtained from the sequential calibration.