Delayed Outcomes in Stagewise Testing

Why Outcome Timing Matters

In a seamless phase 2/3 trial, the transition decision may be ready before every phase 2 outcome is known. A participant may need to complete a fixed follow-up period before a binary response can be assessed; another participant may have completed that period but still be awaiting data entry. If a decision is made using whichever outcomes happen to be available, treatment groups can be compared at different stages of follow-up or with different amounts of information. The resulting evidence may be unstable, and a treatment can be selected because its outcomes matured sooner rather than because its effect is more promising.

The design therefore needs a rule for when outcomes count at the transition and how evidence from each stage enters the final test. A cutoff is not merely an administrative date: it determines which observations contribute to selection, what information the stagewise test represents, and whether the planned error-rate control applies. This tutorial focuses on those timing rules and their consequences. It does not replace the earlier material on handling missing outcomes at the stage transition; the key distinction here is that an outcome still awaiting its defined assessment is not yet an observed failure.

Specify the Cutoff Before Outcomes Accumulate

A useful cutoff rule identifies the eligible participants, the required outcome window, and the data-lock procedure. For a fixed-time binary endpoint, one rule might include every participant randomized by a stated enrollment date who has completed the full 12-week assessment window by the data lock, provided the assessment has been entered and checked according to the protocol. A protocol should also state how it handles assessments received shortly after the lock. Applying an unplanned extension only when it changes a promising result risks making the analysis depend on the observed data.

The timing rule should be applied consistently across treatment groups. That does not mean the same number of outcomes will always be mature in each group: staggered enrollment or operational delays can produce different counts. It means that eligibility is determined by the same prespecified criteria, rather than by whether a particular treatment’s estimate looks favorable. Before interpreting stagewise results, inspect the number randomized, the number with mature outcomes, and the reasons outcomes remain pending in each group.

Maturity is not the same as independence or noninformative observation. If the chance that an outcome is available by the cutoff depends on prognosis, treatment response, or an event related to the endpoint, analyzing only available outcomes can distort a treatment comparison. A fixed cutoff alone does not repair that problem. The design must use an analysis appropriate to the endpoint and observation process, and the selection rule must not silently treat pending observations as failures or discard them as if they were unrelated to outcome.

Stagewise Evidence and Combination Testing

A stagewise test summarizes evidence from a defined stage in a p-value. In a two-stage design, let \(p_1\) be the stage 1 p-value and \(p_2\) the stage 2 p-value for a specified null hypothesis. A common combination method transforms these p-values to normal scores and takes a weighted sum. The transformation and its calibration depend on assumptions; those assumptions must be stated rather than inferred merely from calling the p-values “valid.”

For the usual normal calibration, assume that under the null the stagewise p-values are independent and exactly uniform on \((0,1)\). Then each \(Z_i\) is standard normal, the two normal scores are independent, and

Thus \(Z_C\) is standard normal under the null, and comparing it with the chosen one-sided normal critical value controls the type I error at the corresponding level. Exact uniformity is stronger than validity: a valid p-value may be conservative, with \(P(p_i\leq t)\leq t\) rather than equality for every \(t\). For example, a p-value that is always 0.5 is valid under the null but does not have a standard normal transformed score.

The combination test can still be conservative when stagewise p-values are independent and super-uniform. The rejection event \(w_1\Phi^{-1}(1-p_1)+w_2\Phi^{-1}(1-p_2)\geq c\), for nonnegative weights, becomes less likely when either p-value increases. Independent super-uniform p-values are stochastically no smaller than independent uniform p-values, coordinate by coordinate. Consequently this decreasing-in-p rejection rule has null rejection probability no greater than its probability under independent uniforms. This argument supports calibration; it does not make dependent p-values independent.

The stagewise p-values must be valid for the analyses actually performed, and their independence or independent-increments property must be justified by the design. For instance, using genuinely separate stage data can support independence under an appropriate model. Reusing participants or controls, or allowing stage 2 data to overlap stage 1 data, requires a method that accounts for the dependence. If the treatment was selected using stage 1 results, selection and multiplicity also matter: simply treating the selected treatment’s unadjusted stage 1 p-value as if the treatment had been specified in advance does not establish validity. The earlier material on confirmatory validity after treatment selection and multiplicity across candidate treatments addresses those selection adjustments.

How Delayed Outcomes Can Change Selection

A transition rule often ranks candidate treatments using phase 2 evidence. When outcome maturity differs, the observed proportions can be based on different sample sizes and potentially different enrollment cohorts. Even with the same endpoint definition, a ranking based on available outcomes may differ from the ranking after all planned assessments mature. This is a reason to prespecify a minimum information requirement, a common cutoff, or a rule to wait for additional outcomes—not a reason to assume that the most mature group has the best treatment.

Operational Checks and Design Choices

Before using a phase 2 snapshot, examine maturity by treatment and, where relevant, by site, enrollment period, and important baseline groups. A large difference in completion proportions, a cluster of pending outcomes in one arm, or a change in the reasons for delay can warn that the snapshot is not comparable. Check that the number of mature assessments agrees with the cutoff definition, and document whether records arriving after the lock were excluded consistently. These checks diagnose implementation and possible selection concerns; they do not prove that the remaining outcomes are unbiased.

Two broad alternatives are to wait for a more complete phase 2 data set or to proceed at a prespecified interim cutoff. Waiting reduces uncertainty due solely to immature assessments, but delays the transition and may not eliminate outcome-related observation problems. Proceeding sooner can preserve operational efficiency, but the selection rule must acknowledge the smaller and possibly uneven information base. A third choice is to use a planned stagewise combination test: this can use evidence from both stages, but only with valid stagewise p-values, justified dependence assumptions, prespecified weights and cutoffs, and appropriate adjustment for selection and multiplicity.

These approaches answer different questions. A phase 2 ranking rule chooses which treatment advances; a confirmatory test evaluates a prespecified claim with controlled error. A promising selection statistic is not automatically confirmatory evidence. Nor does combining stagewise p-values automatically correct an outcome-maturity bias in the stage 1 data. The selection rule, outcome timing, and final testing procedure must fit together as one design.

Common Pitfalls and Limitations

• Calling pending outcomes failures. A participant who has not reached the assessment window has no mature response status under that endpoint definition. Coding the outcome as a failure changes the estimand and can bias the comparison.
• Assuming a valid p-value is exactly uniform. Validity permits conservatism. The normal-score mean-zero, variance-one derivation requires exact uniformity; with independent super-uniform p-values, calibration instead follows from the monotonicity argument.
• Using the selected treatment’s unadjusted stage 1 evidence. Selection favors candidates with unusually favorable results. The confirmatory procedure must account for that selection; independence of new stage 2 data alone does not validate an unadjusted combined analysis.
• Assuming a calendar cutoff guarantees comparability. Different outcome maturity can be associated with enrollment timing, prognosis, or treatment-related events. A common date does not establish noninformative observation.
• Changing the cutoff after seeing results. Adding only late-arriving outcomes that help a preferred conclusion makes the analysis outcome-dependent. Define the data-lock rule prospectively and apply it consistently.

A central limitation is that no cutoff rule can recover information that has not yet been observed or, by itself, resolve informative delays. The design must state what the stage 1 data represent, how uncertainty affects selection, and which inferential guarantees apply. The detailed treatment of time-to-event behavior belongs to the later material on nonproportional hazards; here the general lesson is to avoid assuming that an event-driven or maturity-driven snapshot has the same properties as a fixed, fully observed analysis.