What Makes a Data Review an Interim Look?
During a trial, data accumulate before every participant has completed follow-up. A team may check progress, review safety, examine an efficacy result, or make a decision about continuing. These activities are not all the same. For repeated significance testing, the key question is not whether a meeting is called an “interim analysis.” It is whether accumulating trial data are assessed in a way that can affect an inferential conclusion or a decision that depends on the treatment result.
This definition is functional. A formal look might include a prespecified test and a stated decision rule, but a review can still be a look without computing a p-value. Seeing treatment-group outcome summaries, for example, could affect recruitment, follow-up, or the choice of an analysis. Conversely, a routine data-quality check need not be an efficacy look if it does not reveal treatment comparisons or otherwise influence an efficacy decision.
The earlier material on optional stopping and repeated significance testing establishes the central error-control issue: when a true null hypothesis \(H_0\) is tested repeatedly, the probability of at least one false rejection can differ from the significance level \(\alpha\) assigned to a single test. The relevant procedure includes not only the test statistic and threshold, but also when results are examined and what happens after each examination.
Scheduled and Result-Triggered Looks
A scheduled look is planned to occur at a stated time or information point, such as after a specified number of participants have outcome data. Scheduling makes the opportunity to examine the results explicit, but it does not by itself control the overall false-positive probability. That depends on the full testing and decision procedure.
A result-triggered look occurs because some observed result or signal prompts another assessment. The trigger might be an unusually promising interim estimate, a concerning safety pattern, or a threshold reached in an ongoing monitoring display. It can be formal if the trigger rule is specified, or informal if people decide ad hoc to inspect further. A triggered analysis is not “free” of error-control consequences merely because its timing was not on a calendar.
A look can be counted even if it does not lead to a rejection. It is the opportunity to assess the accumulating evidence, not the eventual outcome of the assessment, that matters. Also, the relevant unit is not every time a computer refreshes a dashboard. A refresh is not an inferential look if no treatment-effect information is accessed or used. But if someone repeatedly checks a treatment comparison and acts on what appears, those checks are substantively relevant even if the team calls them informal.
Worked Example: Scheduled Looks and a Stopping Rule
Hypothetical example. A trial plans analyses after 40, 80, and 120 participants have outcome data. At each planned look, the team would compare treatments using a nominal two-sided significance threshold of \(0.05\), and it would stop and reject \(H_0\) at the first look with \(p<0.05\). The values available along a hypothetical path in which the trial continues are \(0.22\), \(0.031\), and \(0.012\), respectively.
The first scheduled look, at 40 participants, has \(p=0.22\), so the rejection rule does not stop the trial. At the second look, \(p=0.031<0.05\), so the stated rule stops the trial and rejects \(H_0\). Thus the third value, \(0.012\), is a potential value under hypothetical continuation, not an observed result under the stated stopping rule. Reporting it as observed would contradict the rule unless the trial actually continued despite the rejection.
There are two scheduled opportunities in the realized path before stopping. The third planned occasion is part of the design, but is not reached in this path. Error control must reflect the planned monitoring and stopping procedure, rather than treating the second-look p-value as if it had necessarily been the only test. This example identifies the looks; it does not calculate the trial’s overall type I error rate.
Why Informal Checks Still Matter
A formal label does not determine whether a check can affect error control. Suppose a researcher sees accumulating efficacy results, continues when they are unconvincing, and stops when they look compelling. Even if no protocol-defined interim test was performed, the decision rule depends on the observed results. The chance of a false-positive conclusion is therefore a property of the full process, including those informal choices.
A short justification is to describe false rejection as an event. Let \(R_j\) denote rejection at the \(j\)th result-based look that is actually reached. The overall false-positive event under \(H_0\) is that at least one such rejection occurs:
The probability of this union depends on which looks are reached and on how their results are used. If a result triggers another analysis, the trigger is itself part of the process that determines which rejection opportunities occur. Consequently, checking only the final reported test can miss earlier opportunities or choices that shaped the final analysis. The union bound gives \(P(\bigcup_j R_j)\leq\sum_j P(R_j)\), but the individual events may be dependent, and the bound is not generally an exact overall error rate.
This argument does not say that every data check increases the false-positive probability by the same amount, or that any access to data automatically invalidates a study. It says that a result-dependent decision path must be considered when describing and controlling the chance of a false rejection. A valid analysis needs a clearly defined monitoring process, not merely a record of the test shown in the final report.
Worked Example: Repeated Informal P-Value Peeking
Hypothetical example. During a trial, an analyst checks the same treatment comparison at the end of each of four weeks. The cumulative-data p-values along the observed path are \(0.42\), \(0.18\), \(0.08\), and \(0.047\). The analyst stops the trial and reports efficacy as soon as a p-value is below \(0.05\).
At the first three checks, the p-values do not meet the analyst’s reporting threshold. At week four, \(0.047<0.05\), so the analyst stops and rejects \(H_0\). The four checks are result-based opportunities, although they were not formally scheduled as interim analyses. The reported \(0.047\) is the value at the stopping look; it is not, by itself, evidence that the overall false-positive probability of this four-check procedure is \(0.05\).
These are cumulative-data tests, so their statistics are dependent; the four p-values must not be treated as if they came from four independent samples. The example does not use an independence calculation. Its point is that the informal stopping rule depends on repeated examination, and the error rate belongs to that complete rule. A nominal single-test threshold cannot be interpreted as the procedure-wide error rate without accounting for how the checks and stopping decision operated.
Operational Review Versus Inferential Monitoring
Trials need frequent operational review. Teams may check whether forms are complete, follow-up is overdue, eligibility criteria were applied, or data appear internally inconsistent. These checks can be essential for trial quality without being looks at treatment efficacy. The distinction depends on what information is visible and what decisions it can influence.
Worked Example: A Data-Quality Check and a Committee Review
Hypothetical example. A study team receives a weekly report showing the total number of enrolled participants, missing outcome forms, and pooled event counts, with treatment assignments hidden. Separately, an independent monitoring committee receives unblinded treatment-group outcomes at a planned review and can recommend stopping for compelling evidence of benefit or harm.
The weekly report, as described, is an operational check: it does not reveal a treatment comparison and is not used to make an efficacy claim. On those facts, it is not an efficacy interim look. If the report were changed to show outcomes by treatment group, or if staff could infer the comparison and use it to alter the efficacy analysis, the classification would need reconsideration.
The committee’s review is an interim look. It has access to unblinded treatment-group outcomes and can recommend a decision based on them. It remains a look even if the committee communicates only a recommendation and does not calculate or report a conventional p-value. Its result-based decision role makes it relevant to the trial’s inferential process.
This distinction prevents two opposite errors: counting every administrative data check as a repeated efficacy test, and ignoring an influential unblinded review merely because it is called “safety monitoring” or does not produce a reported p-value. The actual access, use, and decision authority determine the classification.
Recognizing Looks and Their Limits
When deciding whether an activity is a look, ask what was available to the people conducting it, whether treatment differences could be learned, and whether the information could affect stopping, continuing, enrollment, analysis, or claims. These questions are more useful than asking whether a meeting was formally named an interim analysis.
- Scheduled formal analysis: a planned examination of accumulating treatment results under a stated role or rule.
- Result-triggered analysis: an additional examination prompted by an observed result or signal, whether the trigger was written down or improvised.
- Informal peeking: access to accumulating treatment evidence that can influence decisions, even without a declared test.
- Operational review: a check that does not reveal or use treatment-effect information; it need not be an efficacy look.
There are limits to what classification alone can accomplish. Calling an examination a “look” identifies its relevance; it does not calculate the error rate, establish that a procedure is invalid, or show how much a particular look changes error. The dependence between looks, their timing, the test statistic, the decision thresholds, and the possibility of result-triggered analyses all matter. A fixed-sample final analysis has no interim efficacy looks, while a monitoring procedure deliberately examines accumulating evidence. A continuously viewed display may create repeated opportunities for action, even if it does not have discrete meeting dates. The detailed comparison of monitoring schedules and continuous monitoring requires specifying those procedures.
A useful diagnostic is to compare the documented analysis plan with what actually happened. Unrecorded efficacy dashboards, extra p-value calculations, treatment-specific summaries shared with decision-makers, or analysis changes made after seeing results are reasons to investigate whether additional looks occurred. They do not, on their own, quantify the impact. The key is to reconstruct the information and decisions available at each point, including who saw the results and whether a result could change the trial’s course.
Check Your Understanding
Answer each question by classifying the review and explaining the relevant error-control issue.
- A team checks treatment-specific p-values every Friday but has no written interim-analysis plan. Are these looks?
- A blinded coordinator checks weekly for missing forms and reports only pooled enrollment totals. Is this necessarily an efficacy look?
- A committee sees unblinded outcome summaries and can advise stopping, but never calculates a p-value. Does its review count?
- A planned second look is reached only if the first analysis is not significant. Is the second look irrelevant to error control because it is conditional?
Solutions. (1) Yes. Repeated treatment-specific p-value checks can influence decisions, so the absence of a written plan does not make them irrelevant. (2) No, not on the stated facts: the review is blinded, operational, and does not disclose treatment differences. If access or use changes, reassess the classification. (3) Yes. The committee can use accumulating treatment evidence to influence a decision, and a conventional p-value is not required for an assessment to be a look. (4) No. The rule determining whether the second look occurs is part of the monitoring procedure; error control concerns the probability of rejection over all reached looks and the decisions that follow them.