Why the Timing of a Trial Change Matters
Adaptive trials allow some aspects of a study to change while it is underway. The important regulatory distinction is not simply whether a change occurred after enrollment began. It is whether the change was specified before the relevant trial data could influence the decision, and whether the statistical interpretation accounts for the design that was actually used. A protocol amendment made before anyone examines interim treatment results can still be prospective with respect to those results. Conversely, a change may be post hoc even if it is written into a document shortly after an interim analysis.
This distinction affects how readers assess the evidence. A prospectively planned adaptation can be evaluated as part of the design: its rationale, decision process, and effect on inference can be considered before the outcomes are known. A post hoc change raises a different question: did knowledge of the observed data influence which analyses, endpoints, doses, or hypotheses were selected? That possibility can make a nominal confirmatory result harder to interpret, even when the change was clinically reasonable.
The classification is about the decision process, not the quality of the decision by itself. A planned adaptation may be poorly justified or statistically uncalibrated. A post hoc change may be necessary to protect participants or respond to new information. Neither label alone determines whether the trial's evidence is valid; each helps identify what needs to be evaluated.
Classifying a Change
To classify a change, identify four things: what changed, when the decision was made, what information was available to the decision-makers, and whether the change could affect the inferential claim. Relevant changes may concern treatment arms, sample size, endpoints, the target population, the analysis, or trial conduct. The same operational action can have different classifications depending on whether its decision rule was fixed in advance or chosen after viewing results.
Blinding and access controls matter because they help establish what decision-makers could know. For example, a committee may review interim safety information while remaining unaware of comparative efficacy results. A safety-based action can therefore be planned and implemented without using unblinded efficacy data. But a label such as “blinded review” does not settle the issue on its own: the relevant question is what information was accessible, including indirect clues that could reveal comparative outcomes.
Prespecification also has a statistical meaning. It gives reviewers a chance to judge whether the planned design supports the intended error control and interpretation. Earlier material on conditional error and type I error under data-dependent changes explains why a change made after observing interim data can affect the distribution of a final test. A change being written down beforehand does not, by itself, establish that the final test is calibrated.
Worked Example: A Planned Futility Decision
Suppose a hypothetical two-arm trial plans one interim review after 120 of a target 240 participants have completed the primary outcome. Before any interim outcomes are examined, the protocol states that an independent committee will recommend stopping for futility if the conditional power, calculated using a specified model and the planned final analysis, is below 10%. The sponsor and investigators will not see comparative interim efficacy results.
Classification. This is a prospectively specified adaptation. The review time, decision criterion, analysis basis, and information barrier were set before the relevant comparative results were examined. The decision is data-dependent, but data-dependence does not make it post hoc when the rule governing the decision was set in advance.
Interpretation. The plan supports a more credible account of why the trial stopped than a rule selected after seeing interim results. It does not automatically establish that the overall type I error is controlled or that every implementation detail is sound. Those conclusions require evaluation of the planned test and its assumptions. For example, the conditional-power calculation must use the stated model, and the final analysis must follow the design for which its error properties were assessed.
Why Post Hoc Selection Can Change the Evidence
A simple calculation shows why choosing an analysis after seeing results can make a nominal p-value misleading. Suppose two endpoint test statistics, \(Z_1\) and \(Z_2\), are independent under the global null hypothesis and each has a standard normal distribution. If an analyst chooses whichever endpoint has the larger absolute statistic and then compares that statistic with the usual two-sided 5% cutoff, the chance of at least one false rejection is greater than 5%.
This calculation assumes independence, standard normal statistics under the null, and selection of the larger absolute value. It is a deliberately simple illustration, not a model for every adaptive trial. Under these assumptions, each test separately has a 5% chance of exceeding the two-sided cutoff. The probability that neither exceeds it is the product of the two probabilities of not exceeding it.
Thus the chance of at least one false positive is 9.75%, not 5%. The issue is not that the normal cutoff is incorrectly calculated for either endpoint considered alone. The issue is that the reported test was selected from two opportunities using the observed data. A confirmatory interpretation must account for the selection or be limited appropriately.
Worked Examples: Similar Changes, Different Status
Worked Example: Increasing Enrollment After Interim Results
In a hypothetical trial, investigators planned to enroll 300 participants. The protocol specified that enrollment could increase to 400 if a blinded pooled estimate of the outcome variance exceeded a stated threshold at an interim review. At that review, the pooled variance estimate is 1.44, above the prespecified threshold of 1.30. The rule calls for the increase, and comparative treatment results remain concealed from the decision-makers.
Classification. The sample-size increase is prospectively specified: the trigger, data source, and resulting action were set before the interim review. The value 1.44 is an observed result, but using observed information does not make the decision post hoc if the decision procedure was fixed in advance and the relevant information access matches the plan.
What still needs evaluation. The increase is credible as an implementation of the stated rule only if the variance estimate was calculated as planned and no unplanned comparative information influenced the decision. Statistical validity is a separate question. Reviewers need to know whether the final testing procedure remains calibrated under this sample-size adaptation; a blinded variance review does not itself prove that it does.
Worked Example: Choosing an Endpoint After Seeing Results
Consider a hypothetical trial with two prespecified candidate endpoints. At the end of the study, the observed test statistics are \(Z_1=2.10\) and \(Z_2=1.70\). After examining both results, the team decides to present endpoint 1 as the primary basis for a confirmatory claim. Assume each statistic would have a standard normal distribution under its endpoint's null hypothesis, and use a two-sided 5% cutoff of 1.96.
Calculation for endpoint 1. Its statistic exceeds the cutoff: \(2.10>1.96\). Its two-sided p-value is approximately \(2[1-\Phi(2.10)]=0.0357\), where \(\Phi\) is the standard normal cumulative distribution function.
Classification and interpretation. If the endpoint was elevated to primary status because its observed result was more favorable, that choice is post hoc. The p-value of approximately 0.0357 describes the endpoint-specific test under its stated null, but it does not account for choosing between endpoints after seeing both results. Under the independence assumptions in the preceding calculation, the chance of at least one false positive across two such tests at the 5% level is 9.75%. Therefore, the endpoint-specific p-value alone does not support an unadjusted 5% confirmatory claim that ignores the selection.
Worked Example: Responding to New Safety Information
In a hypothetical trial, an external safety alert about the study drug becomes available before the planned interim efficacy review. The sponsor pauses enrollment while a safety committee assesses the alert. No comparative efficacy results have been accessed. The trial team then proposes to remove a high-risk subgroup from further enrollment.
Classification. The pause may be a response to new external information rather than a post hoc response to trial outcomes. However, the proposed subgroup change was not necessarily prospectively specified. If it was chosen after the safety alert, it is a post hoc operational change relative to that information, even though the alert did not come from an interim efficacy analysis.
Interpretation. A post hoc change may be justified to protect participants, but its implications for the target population and evidence need clear evaluation. If the final efficacy claim is intended to apply to the original population, excluding a subgroup after the trial begins may affect that interpretation. The team should distinguish the safety rationale for the change from any later claims about efficacy, and should not imply that an unplanned population change was part of the original design.
Regulatory Credibility and the Main Alternatives
A prospective plan gives reviewers a basis for judging the decision before knowing its outcome. This reduces concern that the design was tailored to favorable results and makes it easier to compare the planned analysis with what was actually done. Credibility is strengthened when the decision-makers' access to data is consistent with the plan and when the claimed inference is supported by the design's statistical properties.
A fixed design is the main alternative: it specifies the trial's conduct and primary analysis without planned data-dependent changes. It may be simpler to explain, but it is not automatically superior if an adaptation is scientifically justified and properly evaluated. A planned adaptive design can be credible, but requires the adaptation and inferential approach to be considered together.
A post hoc analysis is another alternative, but its role should be described honestly. It can be useful for generating hypotheses, understanding unexpected results, or assessing sensitivity to analytic choices. It should not be presented as if the choice had been made before the relevant data were known. When post hoc decisions affect a proposed confirmatory claim, the analysis may require adjustment for selection or other design changes; in some settings, an independent confirmatory study may provide stronger evidence.
| Approach | What supports credibility | Interpretive caution |
|---|---|---|
| Fixed design | One planned conduct and analysis strategy | Can be inflexible; validity still depends on appropriate methods and conduct |
| Prospectively specified adaptation | Decision criteria set before relevant results are examined; analysis evaluated for the adaptive design | Prespecification alone does not guarantee error control |
| Post hoc change or selection | Transparent rationale and an analysis that addresses the selection or limits the claim | Unadjusted nominal results may overstate confirmatory evidence |
Practical diagnostics include comparing the dated plan with the time of data access, checking whether the stated rule matches the action taken, and identifying who could see which results. Look for unexplained differences between planned and actual endpoints, populations, interim timing, or analyses. Such differences are reasons to investigate, not automatic proof of misconduct or invalidity. A change may have a sound clinical or safety rationale while still changing what the statistical evidence can establish.
Common Mistakes and a Short Classification Process
- “The change was written down before the final report, so it was prospective.” That is not enough. Ask whether it was specified before the decision-makers could examine the data that might drive the change.
- “The rule was prespecified, so the type I error is controlled.” Prespecification identifies the procedure; it does not prove the procedure's error properties. The inferential method must be appropriate for the design.
- “Any post hoc change invalidates the trial.” Not necessarily. The change may be ethically necessary or useful for exploratory analysis. Its timing and consequences must be disclosed, and the strength of the claim must match the evidence.
- “Blinded means no information could influence the decision.” Blinding reduces access to particular information but should be checked in practice, including whether indirect information could reveal comparative outcomes.
Specify what aspect of trial conduct or analysis changed and which outcomes or claims it could affect.
Determine when the decision was made and what relevant trial or external data the decision-makers could examine.
Ask whether the decision followed a rule fixed before the relevant information was available. If not, treat it as post hoc relative to that information.
Assess whether the statistical interpretation accounts for the actual design and any data-driven selection; otherwise qualify the claim or treat the analysis as exploratory.
The central lesson is that timing informs credibility, while statistical calibration and transparent interpretation determine what the results can support. A well-planned adaptation is not a substitute for sound inference, and a post hoc change is not a reason to conceal useful evidence. The reader needs enough information to distinguish what was planned, what changed, why it changed, and how the change affects the claim.
Check Your Understanding
For each question, classify the change or identify the relevant interpretive issue. Solutions follow each question.
- A protocol specifies before interim review that a blinded pooled event rate will determine whether enrollment increases. The stated threshold is crossed and enrollment increases as planned. Is this prospective or post hoc? Solution: Prospective, provided the rule was set before the relevant data were examined and the review used the planned blinded information. Whether the final test remains calibrated is a separate question.
- An analyst tests four outcomes, then labels the one with the smallest p-value as primary. What is the main concern? Solution: The primary outcome was selected after observing results. A nominal p-value for the selected outcome does not, by itself, account for the selection and may not support the intended confirmatory interpretation.
- A safety committee stops enrollment after receiving a new external safety alert, without access to comparative efficacy results. Does the post-start timing alone make this an efficacy-driven post hoc adaptation? Solution: No. Timing alone is insufficient. The decision appears safety-driven and not based on comparative efficacy results, though any resulting change to the population or analysis must still be described and its implications assessed.
- Why is “the rule was in the protocol” not sufficient evidence that a planned adaptation supports a confirmatory claim? Solution: A written rule establishes neither that it was followed nor that the statistical method is valid for the design. Reviewers must also assess data access, implementation, and whether the inferential procedure accounts for the adaptation.