Why the Last Test May Not Be the Trial’s Decision
A final analysis can produce a small p-value, but that number alone does not necessarily describe the decision rule used throughout a trial. If the protocol allowed the trial to stop for significance at an interim analysis, the operative question was whether the test crossed its rejection boundary at any permitted look, not only whether the final scheduled test was significant. The path of results and the stopping rule both matter.
This distinction is especially important when a trial stops early. If the stopping rule ends data collection after an interim rejection, the planned final analysis is not performed on the data that would have been collected had the trial continued. A hypothetical final result is not an observed result. Conversely, if the trial continues to its final analysis, a significant final test does not erase the fact that the trial also permitted earlier rejection.
The calculations below use a simple, prespecified one-sided test to make the distinction concrete. They are hypothetical examples, not reports of actual trials. The focus is how to read the trial’s decision under its stopping rule, not how to design a calibrated boundary for repeated testing.
Define the Rule Before Reading the Results
For a one-sided test at nominal level \(\alpha=0.05\), a commonly used fixed-look standard-normal boundary is \(c=1.645\), since an upper-tail standard-normal probability of 0.05 corresponds approximately to that value. This example uses that same nominal boundary at each look so the stopping rule is easy to follow. It does not imply that testing at this boundary repeatedly controls the trial’s overall Type I error at 0.05.
For the worked examples, assume a hypothetical treatment effect \(\mu\) is measured in outcome units, with \(H_0:\mu=0\) and \(H_a:\mu>0\). At each scheduled sample size, assume independent normally distributed observations with known standard deviation \(\sigma=8\) outcome units. Under \(H_0\), the statistic \(Z_k=\bar{x}_k/(8/\sqrt{n_k})\) has a standard normal distribution at that fixed look. These assumptions justify the fixed-look calculation; they do not make the statistics at different looks independent. The look schedule and the stopping rule are assumed specified before the results are examined.
Follow the Observed Path, Not Just the Endpoint
Worked Example: A Final Significant Test After a Nonsignificant Interim Look
A hypothetical trial plans analyses after \(n_1=40\) and \(n_2=80\) participants. It uses the first-rejection rule with \(c=1.645\): stop for significance at the first look with \(Z_k \ge 1.645\). At the interim look, suppose the estimated treatment effect is \(\bar{x}_1=1.20\) outcome units. At the final look, suppose the cumulative estimated effect is \(\bar{x}_2=1.700\) outcome units. Trace the rule and interpret the final test.
At the interim look, the standard error is \(SE_1=8/\sqrt{40}\approx1.265\) outcome units. Thus,
Since \(0.949<1.645\), the trial does not reject \(H_0\) at the interim analysis and continues. The one-sided fixed-look p-value is approximately \(P(Z\ge0.949)=0.171\), where \(Z\sim N(0,1)\).
At the final look, \(SE_2=8/\sqrt{80}\approx0.8944\) outcome units. Therefore,
The one-sided fixed-look p-value is \(P(Z\ge1.901)\approx0.0287\). Since \(1.901\ge1.645\), the trial rejects \(H_0\) at the final look. Under the stated model, the result is evidence against a zero treatment effect in the prespecified positive direction. The statistic and p-value describe the final fixed-look test; because the trial’s rule allowed earlier rejection, they are not, by themselves, a summary of the complete repeated-testing procedure.
This path reaches the final analysis because the interim statistic did not cross the boundary. If the interim statistic had crossed it, the first-rejection rule would have stopped the trial, and this observed final calculation would not exist.
Why a Final-Look Result Can Miss an Earlier Rejection
Consider first the planned sequence of statistics as if data could be collected through every scheduled look. Define \(A\) to be the event that at least one statistic crosses the boundary, and \(F\) to be the event that the final statistic crosses it. If the final look is among the scheduled looks, then every path in \(F\) is also a path in \(A\). But a path can cross at an earlier look and not cross at the final look. Thus \(F\) does not capture all paths that trigger rejection under the first-rejection rule.
The probability of an any-look rejection can be split into the probability of a final crossing and the probability of an earlier crossing without a final crossing:
This is a set relationship, not a calculation of either probability. The size of the difference depends on the joint behavior of the statistics across looks. In particular, one must not treat repeated statistics based on shared data as independent without justification.
When the protocol stops at the first crossing, later statistics on that trial are not observed. The full sequence above is therefore a useful way to define the trial-level event, but it should not be mistaken for a sequence of observed results after stopping. An early crossing is already sufficient for rejection; there is no need to invent or report later values.
Worked Example: The Rule Stops Before the Planned Final Look
In a hypothetical trial with the same two-look plan and boundary \(c=1.645\), suppose the interim statistic is \(Z_1=2.10\). Determine the trial’s decision and what can be said about the final statistic.
The interim statistic crosses the boundary because \(2.10\ge1.645\). Its one-sided fixed-look p-value is approximately \(P(Z\ge2.10)=0.0179\), which is below 0.05. Under the stated rule, the trial rejects \(H_0\) at the first look and stops. The decision is made at look 1.
Because the rule stops the trial at this crossing, participants scheduled to enter after look 1 are not observed in this trial. There is no observed \(Z_2\), final-look estimate, or final-look p-value to report as a result of this stopping rule. A value one might calculate under the counterfactual assumption that the trial continued is not the result of the trial as conducted. Reporting only a final-look test would omit the actual stopping event.
Worked Example: An Earlier Crossing Can Differ From a Hypothetical Endpoint
Suppose a hypothetical two-look data path, if all planned observations were collected, would have \(Z_1=1.80\) and \(Z_2=1.20\). Apply the first-rejection rule with \(c=1.645\), and explain why the endpoint alone would give the wrong account of the rule’s decision.
At look 1, \(1.80\ge1.645\), so the rule rejects \(H_0\) and stops. The fixed-look one-sided p-value at that look is approximately \(P(Z\ge1.80)=0.0359\). If data collection had continued, the stated hypothetical final statistic \(1.20\) would not cross the boundary; its one-sided fixed-look p-value would be approximately \(P(Z\ge1.20)=0.1151\).
Those two numbers describe a potential full data path, not the results observed under the stopping rule. In the trial as specified, the final statistic is unobserved because the first look triggered stopping. If someone looked only at the hypothetical endpoint, they would say “not significant,” even though the actual rule had already rejected at look 1. The example demonstrates why the final test cannot replace the trial’s decision rule.
How to Read a Trial Result Carefully
The appropriate summary depends on what procedure was actually used. A fixed-sample trial with one prespecified final analysis asks whether that final test rejects \(H_0\). A trial with interim opportunities to reject must also account for those opportunities and for the rule governing continuation or stopping. A calibrated repeated-testing procedure uses boundaries or other prespecified rules designed for its stated error-rate objective; its decision is evaluated against that procedure, rather than by treating the final test as the only test.
A nominal p-value at the final look answers a limited question: under the null model, and for that fixed-look statistic, how surprising is a result at least as extreme in the specified direction? It does not state the probability that \(H_0\) is true. Nor does it, by itself, answer the trial-level question “Would this protocol have rejected at any allowed look?” These are related but distinct questions.
Useful checks when reading or reviewing a repeated-testing result include:
- Confirm the number and timing of the analyses allowed by the protocol.
- Identify the rejection boundary or decision criterion applied at each look.
- Find the earliest look at which the observed statistic crossed its boundary, if any.
- Check whether the trial stopped at that crossing and distinguish observed results from hypothetical later results.
- Determine whether the reported p-value is for a single fixed look or summarizes the prespecified repeated-testing procedure.
Common Mistake and Correction
Correction: Trace the prespecified rule across the looks actually allowed. A final p-value describes its final fixed-look test, while a first-rejection rule is triggered by the earliest qualifying look. If the trial stopped early, a later statistic is not observed; if the trial reached its final look, the earlier path still determines whether a rejection already occurred.
The goal is not to discard the final result. It is to place that result in context: state which look it belongs to, whether earlier looks could trigger rejection, and whether the trial actually reached that analysis. A final significant result may be the first rejection, as in the first example, but it is not a substitute for describing the stopping rule that made it the endpoint.
Check Your Understanding
Answer each question using the first-rejection rule described above, with a one-sided boundary of \(c=1.645\).
- A two-look trial has \(Z_1=1.20\) and \(Z_2=1.70\). At which look does it reject, and why?
- A trial has \(Z_1=1.90\) and stops at the first crossing. Is a final-look p-value observed under the stopping rule?
- Explain why the event of a final crossing is contained within the event of any-look crossing.
- Suppose a full potential path has \(Z_1=1.80\) and \(Z_2=1.10\). What does the rule decide, and which statistic is observed after stopping?
- What question does a final fixed-look p-value answer, and what additional question does it not answer by itself?
Solutions. (1) It does not reject at look 1 because \(1.20<1.645\), then rejects at look 2 because \(1.70\ge1.645\). (2) No. The first statistic crosses, so the trial stops before its final look and no final-look p-value is observed. (3) Every path with \(Z_K\ge1.645\) has at least one crossing, namely at look \(K\), so \(F\subseteq A\). (4) The rule rejects at look 1 because \(1.80\ge1.645\), then stops; \(Z_2=1.10\) is only a hypothetical later value, not observed under this rule. (5) It assesses extremeness for the specified fixed-look test under its null model. By itself it does not say whether the trial would have rejected at an earlier permitted look or summarize the trial’s complete stopping-rule decision.