Two Ways Accumulating Data Can Affect a Study
When data accrue over time, investigators may examine them to monitor progress, check data quality, or assess emerging results. Some examinations are formal hypothesis tests: the hypotheses, analysis, timing, and decision rule are specified in advance. Other examinations are informal, such as reviewing a plot, comparing groups, or calculating an unplanned p-value. The distinction matters because both kinds of examination can influence what happens next.
A formal test is not defined merely by whether someone calculated a test statistic. It is defined by a specified inferential procedure and the role that procedure has in the study's decisions. Conversely, calling an analysis “informal” does not make it irrelevant to false-positive risk if investigators use what they see to stop early, change the endpoint, select a subgroup, or decide which result to report.
These categories describe the procedure and its use, not the sophistication of the calculation. A formally specified test can use a simple statistic; an informal inspection can involve elaborate models. Nor are “formal” and “informal” synonyms for “important” and “unimportant.” A plot that prompts an unplanned early stop may have substantial consequences even if no p-value was calculated.
What Makes a Test Formal?
For a formal monitoring test, a protocol should make clear what question is being tested and how the result will guide action. At a minimum, it should identify \(H_0\) and \(H_a\), the outcome and analysis population, the test statistic and its assumptions, the significance level \(\alpha\), the scheduled look or stopping condition, and the action attached to the result. If multiple outcomes, populations, or analyses can determine a claim, those possibilities also need to be considered in the plan.
The assumptions belong to the planned test, not just to a formula. For example, a z test may rely on a statistic whose null distribution is approximately standard normal. If that approximation is poor for the data, or if the analysis population differs from the prespecified one, a planned cutoff may no longer have its intended interpretation. Prespecification cannot make a model correct; it makes the intended procedure identifiable and reviewable.
Informal inspection can include reviewing enrollment or missingness, plotting outcomes by treatment group, examining an estimated treatment effect, or looking at p-values before a planned analysis. Some such reviews are operationally necessary. The important questions are whether the reviewer can see comparative outcome information, whether the review affects a choice, and whether that choice changes the inferential path. A blinded review of pooled data for data quality is not equivalent to an unblinded review that prompts a treatment-effect claim, although both should be described accurately.
Worked Example: A Prespecified Final Test and an Interim Review
Hypothetical study. A trial compares a treatment with control on a continuous outcome measured at 12 weeks. The protocol specifies one two-sided test at the final analysis: \(H_0:\mu_T-\mu_C=0\) versus \(H_a:\mu_T-\mu_C\ne 0\), with \(\alpha=0.05\). Assume the planned final z statistic is adequately approximated by a standard normal distribution under \(H_0\), and that the outcome and analysis population follow the protocol.
At week 6, a team member views an unplanned interim plot. The standardized treatment contrast is \(Z=1.82\). For illustration, the two-sided nominal p-value is
This week-6 review is not the protocol's formal efficacy test: no week-6 rejection threshold or formal efficacy action was specified. The calculated p-value is above 0.05 in any case, but that fact does not make an informal review harmless. If the team changes recruitment or the outcome definition because the plot looks promising, the study's actual conduct now depends on an unplanned examination.
At the planned final analysis, suppose the statistic is \(Z=2.04\). The corresponding two-sided p-value is
Under the stated final-test assumptions and rule, this final result is statistically significant at the 0.05 level. It does not erase the earlier data-dependent decisions. The formal final test and the informal interim review need to be reported as distinct parts of the study's history.
Why Inspection Can Affect False-Positive Risk
A significance level describes the probability of rejecting \(H_0\) under the conditions of a particular test procedure when \(H_0\) is true. It does not automatically describe every decision made after data have been inspected. If investigators examine results and then choose which analysis, population, endpoint, or stopping time to emphasize, the effective decision rule includes that selection.
The basic logic can be shown without specifying a group-sequential boundary. Suppose a decision is made whenever at least one of two candidate tests has a p-value below 0.05. Under a global null, assume for this illustration that the two p-values are independent and each is uniformly distributed between 0 and 1. The chance of no small p-value is \(0.95\) for each test, and independence gives probability \(0.95^2\) that neither is below 0.05. Therefore,
Thus, under these explicit assumptions, the chance of at least one nominally significant result is 9.75%, not 5%. The example is not a general calculation for repeated looks: tests at successive looks commonly depend on shared data, so independence may not hold. It demonstrates the decision-rule point: searching among results and acting on any favorable one differs from relying on a single prespecified test.
The same reasoning applies when the selection is not phrased as “choose the smallest p-value.” Investigators might continue a study after a favorable trend, stop after an unfavorable one, change the endpoint after seeing which looks strongest, or report a subgroup discovered in the data. These choices can make a nominal p-value an incomplete description of the evidence. The relevant inferential question concerns the procedure that generated the reported result, including choices prompted by inspection.
Worked Example: Selecting an Endpoint After Inspection
Hypothetical study. Investigators plan to measure two outcomes, symptom score and time to recovery, but specify symptom score as the sole primary endpoint for the confirmatory claim. At an informal review, they calculate two-sided p-values for both outcomes: \(p=0.0324\) for symptom score and \(p=0.0444\) for time to recovery. For a transparent numerical check, these correspond approximately to absolute z statistics 2.14 and 2.01:
Under the stated plan, the symptom-score result is the one relevant to the primary confirmatory test; if its analysis assumptions hold, its nominal p-value is below 0.05. The second result is not an additional confirmatory finding simply because it is also below 0.05. If investigators instead decide, after seeing both results, to promote whichever endpoint looks more favorable, they have changed the decision procedure. Treating the selected endpoint's nominal p-value as though it came from an isolated prespecified test ignores that selection.
The distinction is about the path to the claim, not about whether the outcomes are clinically meaningful. Time to recovery may be important, but the protocol's stated role for that outcome and the data-dependent decision to elevate it must be reported. A post hoc finding can motivate further research; it should not be presented as if its status were prespecified.
Keep Formal Testing and Inspection Separate in the Record
A practical way to avoid confusion is to keep two records. The monitoring plan documents formal tests and their decision rules. A review log documents other examinations of accumulating data, including who saw the results, what was examined, and whether a decision followed. This does not turn every plot into a hypothesis test. It makes the actual sequence of information and decisions visible.
Specify the hypotheses, endpoint, analysis, timing, significance level, and action before the relevant results are examined.
Record whether it was a planned inferential test, an operational or blinded review, or an unplanned examination of comparative outcomes.
Note any changes to stopping, recruitment, endpoints, populations, analyses, or reporting that followed the review.
State which result came from the prespecified test and which arose from exploratory inspection; do not imply that a nominal p-value covers unrecorded selection.
This separation is especially useful when data access is restricted. For example, a committee may review unblinded outcomes for safety or trial conduct while the investigators remain unaware of comparative results. The committee's role, access, and actions should be documented. A review can be appropriate and necessary without being the same thing as a prespecified efficacy test.
Comparison, Diagnostics, and Limits
| Feature | Prespecified formal test | Informal data inspection |
|---|---|---|
| Rule and timing | Specified before the relevant results are examined | May be unplanned or lack a fixed decision rule |
| Primary purpose | Evaluate a stated hypothesis using a defined procedure | Explore, monitor, check data, or inform study conduct |
| Interpretation | Uses the test's assumptions and stated error rate | Does not automatically provide a calibrated significance claim |
| Potential influence | Can trigger the action specified in the plan | Can still change stopping, analysis, or reporting choices |
Several warning signs suggest the distinction has become blurred: a “monitoring” plot is followed by an unplanned efficacy stop; the primary outcome changes after interim results are viewed; only the most favorable of several analyses is highlighted; or reports mention a final p-value without describing earlier outcome access and decisions. These do not by themselves prove that a result is false, but they mean the inferential procedure cannot be understood from the final p-value alone.
Formal testing is not automatically a cure for every problem. A prespecified test can still be poorly matched to the data, applied to the wrong population, or interpreted too broadly. Informal inspection is not automatically improper: data-quality checks and safety reviews can be essential. The limitation is narrower and important: unplanned inspection followed by data-dependent choice is not covered merely by labeling the final calculation “formal.”
When there are multiple formal looks or other sources of selection, the overall decision procedure may need an appropriate error-control plan. The exact thresholds and their calibration are separate questions. For now, the essential discipline is to distinguish the formal tests from all other looks, and to account for how both influenced decisions.
Check Your Understanding
Answer each question, then compare with the solutions.
- A protocol specifies one final two-sided test at \(\alpha=0.05\). At an unplanned interim review, a researcher sees a favorable plot and recommends stopping early. Is the plot a formal test? Why does it still matter?
- Under the independent-null assumptions in the derivation above, what is the probability that neither of two tests has \(p<0.05\)?
- An endpoint not designated as primary has \(p=0.03\) after several outcomes were inspected. What is a careful way to describe this result?
- Name two details that should be recorded about an informal review of accumulating data.
Solutions. (1) No. The plot has no prespecified inferential decision rule, but the recommendation makes the inspection consequential: the study's stopping decision now depends on unplanned outcome information. (2) For each independent test, \(P(p\ge0.05)=0.95\); hence \(P(\text{neither }p<0.05)=(0.95)^2=0.9025\), or 90.25%. (3) Describe it as an exploratory or post hoc result, identify the outcome and the fact that multiple outcomes were inspected, and avoid presenting its nominal p-value as if it arose from an isolated prespecified primary test. (4) Examples include who saw the data, what outcomes or summaries were reviewed, whether the review was blinded, and what decisions followed.