Why the Statistical Analysis Plan Must Be Operational
A group sequential design is not fully specified by naming its stopping boundaries or stating that interim analyses will occur. The analysis team must also know which participants and data enter each analysis, when the data set is considered ready, how the decision statistic is calculated, and what follows from each possible result. A statistical analysis plan (SAP) makes those operational details explicit before the interim results are available.
The protocol generally describes the design and its rationale. The SAP translates that design into analysis instructions: a statistician who did not design the trial should be able to determine which records to use, reproduce the calculation, and apply the prespecified decision rule without making discretionary choices based on emerging results. The introductory material on reporting interim design methods in a protocol addresses the protocol-level description; here the emphasis is on the analysis specifications needed to implement it.
The SAP should be finalized and approved before the relevant data are examined by people who could influence trial conduct or the analysis. If a necessary change is made later, document what changed, when and why it changed, who approved it, and whether interim results were already accessible. A dated, version-controlled record helps distinguish prespecified methods from decisions informed by the data.
Define the Analysis Populations and Data
An analysis population is a precisely described set of participants or observations to which a specified analysis applies. Avoid relying on labels such as “intent-to-treat” or “per-protocol” alone: implementations of those labels can differ. State the inclusion rules, the treatment-group assignment used, and any exclusions, including when each exclusion is determined.
For example, a primary efficacy analysis might include every randomized participant, analyzed in the group assigned at randomization. If this is the rule, state it directly and define how outcomes are handled when they are unavailable. A safety analysis might instead include participants who received at least one dose, grouped by treatment received. A per-protocol analysis requires prospective, objective criteria for protocol deviations; an exclusion should not depend on whether a participant’s outcome favors a treatment.
Also specify the outcome definition, assessment window, units, and source variables. State the primary model and the treatment contrast, including covariates and any rules for transformations, stratification, or variance estimation. Missing-data methods should be described rather than left to analyst choice. Naming an analysis population does not, by itself, solve missing outcomes or establish that the analysis estimates the intended treatment effect.
Worked Example: Making the Primary Population Reproducible
Hypothetical trial. A randomized study compares a new treatment with control on a continuous symptom score measured at week 12. The primary analysis is intended to compare groups as assigned at randomization.
A reproducible SAP rule could read: “The primary efficacy population includes all randomized participants. Participants are analyzed according to randomized assignment, regardless of treatment received or discontinuation. The outcome is the week-12 symptom score, with lower scores indicating fewer symptoms. The primary contrast is the adjusted mean difference, new treatment minus control, estimated by the prespecified model using baseline score and randomization strata. Missing week-12 scores will be addressed using the prespecified multiple-imputation procedure.”
This wording identifies who is included, how assignment is handled, what outcome and contrast are analyzed, and how missing outcomes are treated. The analysis still depends on the stated model and missing-data assumptions; those assumptions and sensitivity analyses must be specified elsewhere in the SAP. Simply writing “analyze by intent-to-treat” would not settle these implementation choices.
Specify Interim Data Cutoffs and Information Targets
An interim analysis is based on a defined data set, not merely on a calendar date. The SAP should state the planned look—for example, a target number of primary outcome events or a target information fraction—and how the analysis data set will be assembled. Define the cutoff rules for randomization, outcome observation, data entry, query resolution, and any required follow-up. State which records qualify if an event is reported near the cutoff or if a participant has not completed the assessment window.
When information rather than enrollment determines a look, state how information is calculated and whether the boundary is evaluated at the planned target or at the information available when the data are frozen. Specify the permitted timing tolerance and what happens if the target is missed or exceeded. Do not allow an undocumented choice between using a planned boundary and recalculating a boundary at the observed information: that choice can change the decision rule.
Operationally, record the cutoff date and time, data snapshot or database version, information calculation, unresolved-data policy, and approvals for release of the interim data. Define who may access treatment-specific interim results and who communicates the stopping recommendation. Such controls help prevent knowledge of unblinded results from influencing routine data handling or trial conduct.
Worked Example: An Information-Based Interim Cutoff
Hypothetical design. A trial plans its first interim look at 50% of the maximum information, based on the primary endpoint. The SAP defines information using the inverse variance of the prespecified treatment-effect estimator and requires the endpoint data used at the look to be observed by the cutoff date.
Suppose the data snapshot yields estimated information of 48 units, while the planned maximum information is 100 units. The observed information fraction is
If the prespecified rule permits a look within 0.02 of the 0.50 target and uses the boundary calculated at the observed information fraction, this look is eligible because \(0.48\) is within that tolerance. The SAP must state both conditions; the analyst should not infer them from the fact that the value is “close” to 0.50. If the design instead requires exactly the planned look or prescribes another response to a missed target, that rule governs.
The calculation is meaningful only if the information measure is defined consistently with the estimator and the data cutoff. A change in endpoint availability, model, or variance calculation may change the information and therefore the applicable boundary. The SAP should identify who verifies the calculation before the decision is made.
Write the Decision Calculation as an Executable Rule
For each look, specify the analysis model, test statistic, direction of benefit, boundary, and action. State whether the test is one-sided or two-sided, how the standard error is obtained, and how ties at a boundary are handled. If a design uses alpha spending, identify the prespecified spending approach and the rule connecting information time to the critical value. The SAP should refer to the version-controlled design specification or reproduce the values and calculation inputs needed to implement it.
A boundary is meaningful only in relation to the statistic it is designed for. For example, a positive \(Z\)-statistic may indicate benefit when the contrast is defined as treatment minus control and larger outcomes are favorable. If lower outcomes are favorable, either reverse the contrast or state the direction explicitly. Specify how the statistic is computed at each look, rather than saying only “compare the \(Z\)-statistic with the boundary.”
Worked Example: Applying a Prespecified Efficacy Boundary
Hypothetical assumptions. At a planned interim look, two independently randomized groups are compared on a continuous outcome. The prespecified primary model estimates the mean difference as new treatment minus control, and larger values favor treatment. The design has already specified an upper efficacy boundary of 2.80 for this look and a lower futility boundary of \(-0.50\). The analysis uses the model-based standard error, and the boundaries are assumed to have been calibrated under the design’s stated group sequential procedure.
Suppose the adjusted estimated difference is 2.40 outcome units and its standard error is 0.90 outcome units. The statistic is
Because \(2.667<2.80\), the upper efficacy boundary is not crossed. Because \(2.667>-0.50\), the lower futility boundary is not crossed either. Under the stated rule, this look triggers neither efficacy stopping nor futility stopping; the trial continues to the next planned analysis. The estimate is in the favorable direction, but direction alone does not satisfy the efficacy stopping criterion.
The SAP should say what analysis and action apply if the statistic is exactly on a boundary, if the analysis cannot be run, or if information differs from the planned target. It should also identify whether futility stopping is binding or nonbinding and whether any recommendation is subject to a separate oversight process.
A short justification illustrates why boundary documentation cannot be separated from the number of looks. For a single fixed-look test, a critical value may be chosen to control the type I error at that test. Repeating the same fixed-look test does not preserve that error rate: under \(H_0\), the probability of at least one false rejection across looks is the probability of a union of rejection events. In general,
where \(A_k\) is rejection at look \(k\). Group sequential boundaries are chosen jointly across looks to control the specified error rate; the statistics at successive looks are generally dependent because they use accumulating data. Therefore, a SAP must document the number and timing of looks and the boundary-generation convention, not simply repeat a nominal fixed-look threshold at every interim analysis.
Document Multiplicity Conventions
Multiplicity may arise across interim looks, endpoints, treatment comparisons, subgroups, or combinations of these. These are not interchangeable sources of multiplicity. The group sequential procedure addresses repeated looks for a specified hypothesis or family as designed; it does not automatically define how error is controlled across several primary hypotheses.
For each confirmatory hypothesis, state the null and alternative in context, the significance level, sidedness, and the role of that hypothesis in the overall success criterion. If there are multiple hypotheses, specify the control strategy—such as a prespecified allocation of the error level or a closed testing procedure—and the order or conditions for testing. Explain how that strategy is combined with the interim-look procedure. If no confirmatory claim is planned for an endpoint or subgroup, distinguish its analysis from the multiplicity-controlled primary testing.
Worked Example: Two Primary Endpoints and Interim Looks
Hypothetical plan. A trial has two primary endpoints, and success on either endpoint is sufficient for a positive primary conclusion. The familywise type I error target is 0.05, and the SAP specifies Bonferroni allocation of 0.025 to each endpoint. Each endpoint is analyzed at the same planned interim looks using its own prespecified group sequential boundaries calibrated to its allocated level.
Under the global null, let \(A_1\) be the event that the procedure rejects the null for endpoint 1 at any look, and \(A_2\) the corresponding event for endpoint 2. If each endpoint’s group sequential procedure controls its rejection probability at no more than 0.025, then
This union-bound justification does not require the endpoint tests to be independent. It does require that the stated group sequential control for each endpoint is valid at its allocated level and that the success rule really is rejection of either endpoint. The SAP should state that rule, the allocation, and the endpoint-specific boundary procedure. If success instead requires both endpoints, the scientific criterion differs and must be written explicitly; it should not be inferred from the allocation.
Diagnostics, Deviations, and Documentation
Before an interim decision, verify that the actual data set matches the population and cutoff rules, the information target has been calculated as specified, and the implemented statistic and boundary correspond to the same design version. Reconcile participant counts and endpoint availability against the data snapshot. Independent code review or a validated implementation can help detect mismatches, but it cannot repair an ambiguous SAP.
Warning signs include repeated changes to the cutoff after inspecting treatment differences, analysis populations defined using outcome results, unexplained discrepancies between planned and actual information, a boundary applied to a differently defined statistic, or unrecorded changes to endpoint direction or sidedness. If an operational deviation occurs, record its facts and impact, follow the prespecified contingency rule, and obtain appropriate review. Do not silently substitute a more favorable analysis.
The SAP’s value is not its length but the extent to which another qualified analyst can reproduce the planned decisions. It should identify document ownership and version, approval date, references to the protocol and design specification, software or validated calculation source, and the procedure for recording amendments. Its limitations should also be clear: documentation cannot eliminate model misspecification, missing-data uncertainty, or deviations in trial conduct. It makes assumptions and choices visible so that their consequences can be assessed.
Common Mistake: Treating a Label as a Rule
Correction: Define the population with explicit criteria; state the cutoff and information rule; specify the estimator, standard error, direction, and stopping boundaries; and identify the group sequential and multiplicity procedures that control the intended error rate. A decision rule should be reproducible without choosing details after seeing the interim results.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why is “use the intent-to-treat population” not sufficient by itself to define an analysis population?
- A design plans a look at information fraction 0.60. The SAP specifies that a look may occur within 0.03 of the target and that the boundary is selected at observed information. Is a data set with information fraction 0.58 eligible under this rule?
- At an interim analysis, \(Z=2.41\), the upper boundary is 2.50, and the lower futility boundary is \(-0.60\). What action follows if the rule is to stop only when a boundary is crossed?
- Two hypotheses each have a group sequential procedure controlling its rejection probability at 0.02. What upper bound does the union bound give for the probability of rejecting at least one under the global null?
Solutions. (1) The label does not specify which randomized participants are included, how assignment is handled, what exclusions apply, or how missing outcomes are addressed; the SAP must state these rules. (2) Yes. The difference is \(|0.58-0.60|=0.02\), which is within 0.03, and the specified boundary is evaluated at observed information. (3) Neither boundary is crossed because \(-0.60<2.41<2.50\); continue to the next planned analysis under the stated rule. (4) The probability is at most \(0.02+0.02=0.04\), since the probability of a union is no greater than the sum of the individual probabilities.