Writing the Statistical Decision Algorithm

From Design Description to Executable Rules

A seamless Phase 2/3 design is not fully specified by saying that an interim analysis will “select the promising treatment” and then “test it.” The protocol and statistical analysis plan need to define exactly which data are available at each decision, how selection is made, which observations enter each test, what error rate is controlled, and what action follows every possible result. These rules should be fixed before unblinded interim data are examined.

This tutorial develops a complete decision algorithm for a simple but important case: several candidates are compared during Stage 1, one candidate is selected, and a new, independently randomized Stage 2 cohort provides the confirmatory test. The key inferential point is that selection on Stage 1 data does not, by itself, require dividing the Stage 2 significance level by the number of candidates when only the selected candidate can be claimed and the Stage 2 test is conditionally valid. The algorithm changes if multiple confirmatory claims are possible or if Stage 1 evidence contributes to the final test.

Specify the Information and Actions Before Testing

For each stage, identify the data cut and the status required for an observation to be included. A cutoff can be defined by a calendar date, by completion of a specified follow-up window for every randomized participant, or by another operational rule. The rule should also state how data received after the cutoff are treated. For example, a participant with no recorded endpoint at the cutoff must not silently disappear from the analysis: the estimand, analysis population, and prespecified missing-data method determine how that participant is handled.

A selection rule should be reproducible from the specified data. It might rank eligible candidates by a prespecified estimate of treatment effect, subject to a safety criterion and a minimum amount of usable information. “Select the most promising arm” is incomplete unless “promising,” tie handling, eligibility, and the effect of missing or delayed outcomes are defined. Selection is a design action, not automatically a hypothesis test or a basis for a final efficacy claim.

The algorithm must also specify what each test targets. For example, for a binary response where a larger response proportion is better, one possible Stage 2 hypothesis for the selected candidate is \(H_0:p_T\leq p_C\) against \(H_a:p_T>p_C\), where \(p_T\) and \(p_C\) are the Stage 2 response probabilities for the selected treatment and concurrent control. State the test, sidedness, significance level, analysis population, and any covariate adjustment in advance. The test must be appropriate for the endpoint and its assumptions; the particular test used in an application should be prespecified rather than chosen after seeing results.

Why an Independent Stage 2 Test Can Use Its Full Level

Suppose the Stage 1 data determine which single candidate is tested, and the Stage 2 test uses only new data. Let \(S\) denote the candidate selected by the Stage 1 rule. Assume that, for each candidate \(k\), its Stage 2 test is valid at level \(\alpha\) conditional on the Stage 1 selection and on the selected candidate being tested. In particular, under that candidate’s null hypothesis, its conditional probability of rejection is at most \(\alpha\).

If candidate \(k\) is selected while its null is true, a false claim occurs only if its Stage 2 test rejects. The probability of a false claim can therefore be bounded by conditioning on which candidate was selected:

The final inequality follows because the probabilities of selecting candidates whose nulls are true sum to no more than 1. Thus, when exactly one candidate can be claimed and the new Stage 2 test is conditionally valid, selection among candidates using Stage 1 data does not require an \(\alpha/K\) threshold for that single Stage 2 test. Independence of the new cohort is a clear way to support the required conditional validity, but the analysis plan must still ensure that the Stage 2 test itself remains valid under the selection and conduct of the trial.

This result does not mean multiplicity can always be ignored. If the final report may claim efficacy for several candidates tested in Stage 2, the family of possible claims must be controlled. Nor does the result justify treating Stage 1 and Stage 2 evidence as independent when participants or outcomes overlap. When the final test incorporates Stage 1 evidence, the selection and testing procedure must be calibrated together; an ordinary pooled test is not automatically valid after outcome-dependent selection.

Multiplicity When More Than One Claim Is Possible

If the design retains more than one candidate for confirmatory testing, define the family of hypotheses and the familywise error rate (FWER): the probability of making at least one false rejection among that family. If each candidate’s p-value is valid conditional on the Stage 1 decision, a prespecified procedure such as Holm’s step-down procedure can control FWER at the chosen level even when the tests are dependent. A simple, often conservative alternative is Bonferroni: for \(m\) possible claims and familywise level \(\alpha\), reject a hypothesis only when its p-value is at most \(\alpha/m\).

The relevant multiplicity is determined by the claims the design permits, not merely by how many candidates were screened. If one candidate is selected and only that candidate can be claimed, the independent Stage 2 argument above applies. If two candidates can both be claimed, a two-hypothesis family needs a procedure controlling the chance of any false claim. The family and procedure should be specified before outcome data are reviewed.

Cutoffs, Stopping Rules, and Final Reporting

A decision algorithm should be executable in every branch. Include rules for insufficient data, tied selection scores, no eligible candidate, safety concerns, futility, and a successful confirmatory test. Specify who performs the interim review and whether the treatment assignments and comparative results are accessible to trial personnel involved in conduct. Unplanned access or informal changes to the rule can affect trial conduct and undermine the planned inference.

Timing matters when outcomes are delayed. A data cutoff based only on whichever outcomes happen to arrive first can give candidates unequal information or make the selection depend on outcome availability. Define maturity and follow-up windows before the interim analysis, and state how missing or delayed outcomes are handled. The earlier material on delayed outcomes and missing outcomes provides relevant considerations; this algorithm must turn them into operational rules for the particular design.

Stopping rules need equally precise wording. A safety rule might require pausing enrollment when a prespecified safety boundary is crossed, followed by a defined review and possible restart or termination. A futility rule should identify the statistic or criterion, the timing, and whether stopping is binding. If efficacy stopping is allowed before the planned Stage 2 analysis, its effect on the type I error rate must be incorporated into the inferential design; an unplanned early rejection cannot simply be treated as if the planned single test had occurred.

The final report should match the evidence actually used. State the selected candidate, the selection rule, the Stage 2 analysis population, the endpoint and follow-up window, the estimate and its units, the test result, and any multiplicity adjustment. If the primary test fails to reject, report that the trial did not establish superiority under its prespecified criterion; do not say that the treatments are equivalent unless a suitable equivalence design and analysis were planned.

Diagnostics, Alternatives, and Common Pitfalls

Before the interim decision, check whether the specified data cutoff was applied consistently, outcome maturity is comparable across arms, selection can be reproduced from the recorded variables, and all safety and eligibility criteria were evaluated as written. At the final analysis, verify that the Stage 2 test used only the permitted participants and data, the correct selected candidate and concurrent control, and the prespecified endpoint definition. A mismatch between the planned and actual analysis is a signal to assess whether the claimed error control still applies.

The independent Stage 2-only strategy has a clear interpretation and makes the selection argument straightforward, but it does not use Stage 1 efficacy outcomes to increase the confirmatory evidence. An alternative is to combine stagewise evidence in a design that explicitly accounts for the adaptive selection. Such a procedure must specify how stagewise evidence is combined and calibrate the test, including multiplicity where relevant; simply pooling all observations as though the treatment had been fixed from the start is not a substitute. A third option is to retain multiple candidates and control the family of confirmatory claims with a prespecified procedure such as Holm or Bonferroni. Each approach makes a different trade-off between information use, operational complexity, and the claims available.

The central discipline is to align each claim with the data and decision rule that justify it. A design is not made confirmatory by calling Stage 2 “Phase 3”; it is the prespecified testing procedure, valid for the adaptation actually used, that supports the final statistical conclusion.