Why the Rule Must Be Reproducible
A multi-endpoint interim design is not fully specified by listing endpoint names and an overall significance level. The testing rule must say which hypotheses belong to the family, what evidence is assessed at each analysis, how local boundaries are determined, and how the rule changes after a rejection. Otherwise, two analysts could use the same data and reach different conclusions while both claiming to follow the design.
A reproducible specification is an operational algorithm: given the planned design and the results available at a look, it determines which hypotheses are tested, which are rejected, what level remains available, and whether a success or stopping criterion has been met. The introductory material on graphical multiple testing establishes the role of endpoint allocations and transfers. Here the emphasis is on reporting those elements precisely enough that the rule can be independently implemented and audited.
What a Complete Specification Contains
Begin by naming every confirmatory endpoint and its hypothesis. State the population, treatment contrast, outcome definition, effect direction, and estimand for each endpoint; give the null and alternative in words and mathematical form. For example, for an endpoint where a positive treatment-minus-control contrast is beneficial, one might test \(H_{0i}:\theta_i\leq 0\) against \(H_{ai}:\theta_i>0\). Say whether tests are one-sided or two-sided and give the overall familywise type I error target \(\alpha\).
Next state the endpoint family: which hypotheses are controlled together, which are outside the confirmatory family, and whether any ordering or hierarchy constrains testing. Name the planned looks and define their timing. Information fractions should identify the information time used for each endpoint and look, not merely the fraction of participants enrolled. When endpoint maturity differs, state the endpoint-specific data cutoff and whether results are tested at that look or deferred. Give the rule for unscheduled analyses, delayed results, and any change in timing.
For each local sequential test, identify the statistic and its direction, the boundary at every planned look, and how the boundary was obtained. Boundaries may be given as critical values for a test statistic or as local p-value cutoffs; either way, report the scale and the strictness of the comparison. If boundaries depend on information time, correlation, or the testing history, specify the calculation or lookup procedure, including its inputs. A statement such as “use an O’Brien–Fleming-type boundary” is not enough unless it uniquely identifies the implemented boundary values or algorithm.
For a graphical procedure, report the initial level assigned to each hypothesis and every directed transition weight. For each rejection, state which hypotheses receive level, the fraction each receives, and whether any level is retained or discarded. The weights must be interpreted as fractions of the rejected node’s available level, and must be given for all relevant transitions. State how the graph is updated after each rejection and how its current levels determine subsequent local sequential tests.
The rule should also state the trial-level success criterion. “At least one endpoint is significant” is not equivalent to “both co-primary endpoints are significant,” nor to a hierarchy requiring one endpoint to be rejected before another. Define whether success requires rejection of a particular endpoint, a specified combination of endpoints, or a stated number of hypotheses. Say whether early success stops all efficacy testing or only testing for the rejected endpoint. Make clear what happens if the trial reaches its final look without meeting the success criterion.
Why the Reporting Details Matter
A short justification helps explain why a detailed specification is necessary. Let \(R_k\) denote the set of hypotheses rejected by look \(k\), and let \(A_k\) denote the vector of graph levels just before that look. A decision rule must determine the next levels from the current levels and the observed rejections. In a simple representation,
where \(T\) is the pre-specified transition rule. If the report omits a transition weight, the treatment of a partial transfer, or whether rejected nodes are removed, then \(T\) is not uniquely defined. Consequently, the same observed rejection set can lead to different later local levels and different conclusions. Stating the graph update and the boundary-calibration procedure makes the sequence of decisions reproducible.
Validity requires more than reproducible prose. The local tests must be valid for the planned repeated looks, and the multiplicity procedure must control the familywise error under the intended null configurations. If joint calibration relies on endpoint or across-look dependence, the design must state the assumed structure and how it is used. When the procedure is designed to remain valid over a range of dependence structures, report that calibration basis instead. A simulation used to calibrate or stress-test the rule should identify its assumptions, scenarios, and decision algorithm; simulation evidence does not replace a precise written rule.
Worked Example: Specifying a Two-Endpoint, Three-Look Rule
Hypothetical design. A confirmatory trial has two endpoints. A positive treatment effect is beneficial for both. The one-sided familywise error target is \(\alpha=0.025\). The design makes analyses at information fractions 0.50, 0.75, and 1.00 for each endpoint. Initial graph levels are \(\alpha_1=0.015\) and \(\alpha_2=0.010\), with transition weight \(g_{12}=0.60\) and \(g_{21}=0.40\).
Hypotheses and family. The family consists of \(H_{01}:\theta_1\leq0\) versus \(H_{a1}:\theta_1>0\), and \(H_{02}:\theta_2\leq0\) versus \(H_{a2}:\theta_2>0\). Each \(\theta_i\) is the pre-specified treatment-minus-control contrast in the named endpoint’s estimand. The trial meets its efficacy success criterion if either endpoint is rejected under the familywise procedure.
Analysis and boundaries. At each planned look, use the pre-specified endpoint-specific standardized statistic \(Z_{ik}\), oriented so larger values favor treatment. For each endpoint, provide a table of the local critical values \(c_{ik}\), or a fixed algorithm that returns them from the current graph levels and the look’s information times. The hypothetical protocol stipulates that the values are calibrated for the three-look design and its stated multiplicity procedure. A report must include the actual values or the exact versioned algorithm; the phrase “calibrated for three looks” by itself would not be sufficient to implement the design.
Transitions and stopping. After a rejection of endpoint 1, transfer 60% of its available level to endpoint 2; after a rejection of endpoint 2, transfer 40% of its available level to endpoint 1. Specify that the remaining fraction is discarded if it is not transferred. Once a hypothesis is rejected, remove it from further testing. At each subsequent look, apply the calibrated local boundary corresponding to the updated graph and current analysis history. Stop for efficacy if either endpoint is rejected; otherwise continue to the next planned look, unless a separately specified non-efficacy stopping rule applies.
This specification identifies the family, timing, direction of evidence, initial allocation, transfer rules, and trial-level success criterion. To be operationally complete, the protocol must attach the exact local boundary values or uniquely reproducible calibration algorithm and define how it handles unavailable endpoint results and deviations from planned information times.
Worked Decisions and Alpha-Level Updates
A good report distinguishes the amount transferred from the amount that ceases to be allocated. Transfer weights are proportions of a rejected node’s available level. They do not authorize increasing the familywise error target, and a partial transfer does not conserve the total amount currently allocated to active nodes.
Worked Example: Partial Transfer and Discarded Level
Hypothetical allocation. Immediately before a planned decision, endpoint A has available level 0.015 and endpoint B has available level 0.010. The rule says that after rejection of A, one-half of A’s available level transfers to B, and A is then removed. Assume A is rejected at this decision and no part of A’s level has already been spent for this simplified allocation calculation.
The amount transferred is
B’s updated level is \(0.010+0.0075=0.0175\). The other half of A’s level is \(0.015-0.0075=0.0075\), which is discarded when A is removed. Before the update, the total allocated level was \(0.015+0.010=0.025\). After the update, the active allocation is 0.0175, not 0.025. The transfer has not increased the total allocated level; the total has decreased by the discarded 0.0075.
The report should not say that level is “conserved” in this case. Conservation would occur only if all 0.015 were reassigned to active hypotheses. The future boundary for B must be obtained from the procedure specified for its updated level and the remaining looks; simply adding 0.0075 to a nominal cutoff is not a valid boundary update.
Worked Example: Applying a Reported Local Boundary
Hypothetical interim result. At look 2, endpoint B’s pre-specified local boundary is \(c_{B2}=2.45\). Its observed statistic is \(Z_{B2}=2.31\), with a one-sided nominal p-value of 0.0104. The protocol states that endpoint B is rejected at this look only when \(Z_{B2}\geq c_{B2}\); a p-value is reported descriptively and is not an alternative decision criterion.
Compare the statistic with the boundary: \(2.31<2.45\). Therefore, B is not rejected at look 2. The fact that the nominal p-value is below 0.05 does not change the decision. The result can be reported as: “At the second planned analysis, endpoint B did not cross its local efficacy boundary (\(Z=2.31\), boundary \(2.45\); nominal one-sided \(p=0.0104\)).” The report should then state the next action required by the rule, such as continuing to the final look with the specified graph state.
Worked Example: Reporting an Unavailable Endpoint Result
Hypothetical timing issue. A protocol plans to analyze endpoint C at information fraction 0.60 and endpoint D at 0.60. At the scheduled analysis, C has reached its required maturity, but D has not. The protocol’s pre-specified rule is that a hypothesis with an unavailable endpoint result is not tested at that look; its graph level is not transferred merely because the result is unavailable, and its planned analysis is deferred to the next specified data cutoff.
Suppose C’s observed statistic is 2.62 and its local boundary is 2.58. Because \(2.62\geq2.58\), C is rejected at the scheduled analysis. D is not tested: there is no statistic to compare with its boundary. Under the stated rule, D retains its current graph status and level until the next scheduled opportunity; its unavailability is not a rejection and does not trigger an alpha transfer. The written interim report should state the available information time for each endpoint, the reason D was unavailable, C’s boundary comparison, the graph update after C’s rejection, and the planned date or information condition for D’s next test.
This example illustrates why availability rules belong in the testing specification. Another design could defer the entire look or use a different pre-specified convention, but analysts must not improvise that choice after seeing which endpoint is delayed.
Diagnostics, Alternatives, and Limitations
Before an interim rule is used, audit whether its description can be translated into a unique sequence of decisions. Useful checks include:
- For every endpoint, confirm that the hypothesis, contrast, effect direction, and analysis population are unambiguous.
- For every planned look, verify the timing or information fraction, data cutoff, and endpoint availability rule.
- Recalculate graph updates for each possible rejection pattern, including simultaneous rejections, partial transfers, and discarded portions.
- Check that boundary tables or calibration code correspond to the stated family, graph, look schedule, and assumptions about dependence.
- Confirm that local p-values are not being substituted for sequential boundaries and that the same alpha is not spent again at later looks.
- Compare the implemented decision log with the protocol rule, especially after timing changes or unavailable results.
A common mistake is to report only a nominal significance threshold, then describe any \(p<0.05\) result as a rejection. That reasoning ignores both the interim boundary and the current endpoint-specific level. The correction is to report the test statistic and the applicable local boundary (or the exact local p-value cutoff), then make the decision using that rule. A nominal p-value may still be reported, but it does not override the pre-specified criterion.
Another mistake is to say that a graphical procedure “conserves alpha” whenever a rejected node is removed. That is true only when all of that node’s available level is reassigned. With partial transfers, the untransferred amount may be discarded, so the total level allocated to active hypotheses decreases. A report should state whether that portion is discarded, retained, or sent elsewhere; it should never imply that a transfer creates level.
A compact boundary table can be easier to audit than prose, while a version-controlled algorithm may be necessary when boundaries depend on history or joint calibration. A table is preferable when it covers every planned state clearly; an algorithm is preferable when the number of states makes a table unwieldy. Either approach is acceptable only if it is fixed, validated, and sufficiently detailed to reproduce every decision. Neither a table nor code removes the need to state the assumptions and success criteria in interpretable language.
The rule also has limits. It applies to its defined endpoint family, estimands, looks, and calibration assumptions; an unplanned endpoint, changed analysis time, or different statistic is not automatically covered. A pre-specified testing rule does not establish clinical importance, and crossing a statistical boundary is not itself a guarantee of a favorable benefit-risk conclusion. Separate procedures may be needed for safety monitoring or other decisions.
Check Your Understanding
Answer each question using the definitions and rules above. Brief solutions follow the questions.
- A rejected endpoint holds level 0.012 and transfers 75% to another endpoint. How much is transferred, and what happens to the remainder if the rejected endpoint is removed?
- At a look, a reported statistic is 2.40 and the applicable local boundary is 2.50. The nominal p-value is 0.008. Is the endpoint rejected under a rule requiring the statistic to reach the boundary?
- Name four pieces of information needed to reproduce an endpoint’s interim test beyond its endpoint name.
- An endpoint result is unavailable at a planned look. Under a rule that defers its test without transferring its level, what should the interim report say about that endpoint?
- When is the total level allocated to active hypotheses conserved after a rejection and graph update?
Solutions. (1) The transfer is \(0.75(0.012)=0.009\); the remaining \(0.003\) is discarded if the rule says it is not reassigned. The total allocated level then decreases by 0.003. (2) No. Since \(2.40<2.50\), the endpoint does not cross its local boundary; the nominal p-value does not replace that criterion. (3) Any four of the following: the hypothesis and effect direction, statistic, planned look and information time, boundary or exact calibration algorithm, applicable graph level and update history, endpoint data cutoff, and availability rule. (4) State that the endpoint was not tested because its result was unavailable, identify the reason or status and the next planned testing opportunity, and confirm that its level was not transferred under the specified rule. (5) Only when all available level from the rejected node is reassigned to active hypotheses; partial transfer can leave a portion discarded, reducing the total allocated level.