Simulating Terminal Decisions After Arm Selection

From interim selection to terminal decisions

An interim rule determines which treatment arms continue; the terminal rule determines which of those continuing arms meet the prespecified criterion for a positive result. In a simulation, both actions must be represented. For each replicate, the program needs to apply the final test to the correct surviving arms and retain a clear record of each arm’s decision and the trial’s overall result.

The terminal analysis is not an opportunity to choose a more favorable test after seeing the simulated data. The hypotheses, analysis population, test statistic, direction of benefit, and critical value (or equivalent p-value rule) should all be specified in the design being simulated. In particular, a threshold that is appropriate for one sequential design cannot automatically be assumed appropriate for another.

Assumptions and the terminal test

For the calculations below, suppose the final outcome is continuous, larger values are better, and the terminal analysis compares each surviving treatment with a shared control. Assume independent participants across arms, independent outcomes within each arm, and normally distributed outcomes with a common known standard deviation \(\sigma\). The final sample sizes are fixed and positive. Under these assumptions, the sample mean in each arm has variance \(\sigma^2/n\), and the treatment and control sample means are independent. These assumptions give a simple standardized test statistic; a real protocol may instead require a different outcome model or variance estimator.

Under the boundary null \(\theta_i=0\), the statistic has a standard normal distribution under the stated assumptions. A protocol may specify rejection when \(Z_i\geq c_i\), where \(c_i\) is the critical value for that arm and design. Equivalently, the one-sided p-value \(1-\Phi(Z_i)\) may be compared with a prespecified level. The threshold need not be the same for every arm. It may reflect a design’s planned multiplicity or sequential testing procedure; the simulation must implement the rule that was actually specified.

The independence assumptions also explain why statistics for different arms are not necessarily independent when they share the same control. For two distinct arms \(i\) and \(j\), the treatment means are independent of one another and of the control mean, but both treatment-control differences contain the same control mean. Their covariance is therefore \(\sigma^2/n_0\). The corresponding correlation of standardized statistics is $$ \operatorname{Corr}(Z_i,Z_j)= \frac{1/n_0}{\sqrt{(1/n_i+1/n_0)(1/n_j+1/n_0)}}. $$ This dependence does not change how to calculate each statistic, but it is one reason not to infer a trial-wide error guarantee merely from the threshold used for a single comparison.

A reliable implementation sequence

The key programming distinction is between the set of arms that reach final analysis and the set of all arms originally randomized. A dropped arm is not tested at the terminal stage unless the protocol explicitly defines some other analysis for it. A replicate with no surviving arms has no terminal arm tests; it should still contribute to summaries defined for every simulated trial.

This sequence also makes records auditable: a reviewer can identify which arms were tested, which rule was applied, and how the trial-level summary followed from the arm-level results. It is useful to store a distinct status for arms that were dropped, arms that survived and were tested, and replicates with no survivors. A dropped arm is not the same thing as a tested arm with a non-rejection.

Shared-control survivors and trial-level records

When multiple arms survive, apply the rule separately to each arm using the appropriate final data and threshold. Then derive trial-level summaries from the resulting decisions. For example, if the trial-level summary is “at least one terminal rejection,” it equals 1 if any tested survivor is rejected and 0 otherwise. If the design has a different prespecified trial-level endpoint, use that definition instead.

Why the simulation must preserve the protocol rule

A terminal p-value and a terminal decision are related but are not interchangeable records. The p-value is calculated under a specified test’s null distribution; the decision also depends on the prespecified threshold. A program that saves p-values but later applies an improvised threshold has not faithfully implemented the design. Conversely, a program can make the correct decision from the statistic without storing a p-value, but retaining both often helps with auditing and diagnostics.

The principal alternatives are different representations of the same prespecified test, or genuinely different tests. For a one-sided normal test, a critical-value rule \(Z_i\geq c_i\) and the corresponding p-value rule \(1-\Phi(Z_i)\leq 1-\Phi(c_i)\) produce the same decision. They are computational alternatives, not different inferential procedures. A protocol may instead require a different final statistic or a specified multiplicity-adjusted procedure. Such a method cannot be replaced by separate unadjusted comparisons merely because the formula is easier to code.

Selection at interim is particularly important. The terminal simulation should not silently re-test arms that were dropped, nor should it choose thresholds after seeing which arms survived. A single comparison’s nominal level does not, by itself, prove a familywise or other trial-wide error guarantee for the sequential, shared-control design. The full design rule and its calibration must be respected; evaluating overall operating characteristics is a separate simulation task.

Diagnostics and common failure modes

A useful implementation check is to compare, for each replicate, the tested-arm identifiers with the survivor set. They should match exactly: no dropped arm should appear among the terminal tests, and no survivor should be omitted unless the protocol specifies a reason such as unavailable data. For each tested arm, verify that the statistic uses the correct control mean, sample sizes, standard error, direction, and threshold. A small hand-calculated replicate is valuable for confirming these details.

Also check that the trial-level rejection count equals the sum of the arm-level indicators and that the any-rejection indicator is 1 exactly when this count is positive. Test the boundary explicitly: if the rule says \(Z_i\geq c_i\), equality is a rejection; if it says \(Z_i>c_i\), it is not. A mismatch between the comparison operator and the written protocol can change decisions, especially in unit tests or rounded output.

• Wrong reasoning: “Every arm is compared with the control at the same nominal level, so applying that level separately must implement the design.”
• Correction: Use the terminal rule specified for the full design. Shared controls and interim selection can matter to design-wide calibration; do not substitute an unadjusted per-arm rule unless that is what the protocol prescribes.
• Wrong reasoning: “An arm that does not survive is a terminal non-rejection.”
• Correction: It was not terminally tested. Record its interim status separately from the terminal result for a surviving arm.
• Wrong reasoning: “A replicate with no survivors can be omitted from trial-level summaries.”
• Correction: If the summary is defined over all simulated trials, retain the replicate and record its no-survivor status and the appropriate trial-level outcome.

Finally, a mismatch between the chosen variance estimator and the protocol’s analysis, data that do not satisfy the assumed model, or a missing terminal outcome can invalidate a calculation even when the code runs without error. Do not silently replace missing results with favorable values or discard such replicates. Record the issue and follow the protocol’s prespecified handling. The normal known-variance calculation here is a teaching model, not a universal terminal-testing formula.