Scheduled Looks Versus Continuous Monitoring

Two Ways to Create Rejection Opportunities

A monitoring plan can specify a finite set of analysis times, or it can allow the test statistic to trigger a decision at any time in a monitoring interval. These plans create different rejection opportunities. With a finite schedule, a false positive occurs if at least one scheduled test rejects under \(H_0\). With continuous monitoring, it occurs if the statistic crosses a boundary at any time in the interval. In either case, the relevant error rate concerns the whole monitoring rule, not just one test considered in isolation.

This distinction does not mean that continuous monitoring is automatically invalid, or that a finite schedule automatically controls error. It means that the rejection rule must match the opportunities to reject. The introductory material on repeated significance testing establishes why repeated testing can inflate false-positive risk; here we compare the form of the opportunities and clarify what can, and cannot, be inferred just by counting them.

For a finite schedule, write \(A_k\) for the event that look \(k\) rejects under \(H_0\). The any-look event is the union \(A_1\cup\cdots\cup A_K\). If the tests are independent and each has false-positive probability \(\alpha\), the probability of no rejection is \((1-\alpha)^K\), so the any-look probability is \(1-(1-\alpha)^K\). If they are dependent, that independence formula does not apply. A general upper bound is \(P(A_1\cup\cdots\cup A_K)\leq\sum_{k=1}^K P(A_k)\), which follows because the indicator of a union cannot exceed the sum of the indicators of its component events. Dependence affects the actual union probability, even when the individual tests have the same size.

Finite Schedules: Count the Opportunities, Then Account for Dependence

The independent-test formula gives a useful benchmark, not a universal answer. For cumulative analyses of a growing data set, later test statistics share much of the earlier data. Their rejection events are therefore generally dependent. More looks still add possible rejection events, but the size of the increase depends on how those events overlap. A new look can add little risk if its rejection event largely coincides with earlier ones; it can add more when it contributes a substantial chance of rejection on paths that did not reject before.

Continuous Monitoring: A Maximum Over Time

A continuous rule defines a statistic \(Z(t)\) and a boundary, then rejects if \(Z(t)\) crosses that boundary at any time \(t\) in a stated interval. For a two-sided rule with constant boundary \(c\), the false-positive event is \(\{\sup_{t_0\leq t\leq T}|Z(t)|\geq c\}\). The lower limit \(t_0\) matters: a statistic may not be defined, stable, or suitably approximated at the very start of data collection. In many sequential models, standardization at time zero is undefined.

A useful idealized null model is \(Z(t)=B(t)/\sqrt{t}\), for \(t>0\), where \(B(t)\) is standard Brownian motion: it starts at zero, has continuous paths and independent increments, and \(B(t)-B(s)\sim N(0,t-s)\) for \(t>s\). Under this model each \(Z(t)\) has a standard normal distribution, but the values at different times are dependent because they are built from the same process. Continuous monitoring on \([t_0,T]\), with \(t_0>0\), asks whether the maximum absolute value of this entire path reaches \(c\). It is not equivalent to performing infinitely many independent tests.

The finite-schedule and continuous events can be compared directly when they use the same statistic and boundary. If the scheduled times \(t_1,\ldots,t_K\) lie in \([t_0,T]\), every path that crosses at one of those scheduled times also crosses during the full interval. Consequently, the continuous-monitoring event contains the scheduled any-look event, and its probability cannot be smaller. This comparison relies on keeping the process, boundary and monitoring interval the same. A different boundary or a different test definition changes the comparison.

Why the Number of Opportunities Is Not the Whole Story

For fixed marginal test sizes, adding a scheduled look cannot remove an existing rejection event: the new any-look event contains the old one. Thus its false-positive probability cannot decrease. But the number of looks alone does not determine how much it increases. The dependence between looks, their timing, the test statistic, the boundary and the monitoring interval all matter. For continuous monitoring, the opportunities are described by a path and a crossing rule rather than by a finite count.

A tempting but incorrect argument is that continuous monitoring has infinitely many looks, so its false-positive probability must be 100%. The error is treating infinitely many highly dependent evaluations as if they were independent tests with a fixed positive rejection probability. A continuous path's crossing probability is determined by its joint behavior over time and by the boundary; it is not obtained by substituting an infinite number into the independent-look formula. Conversely, using a per-time threshold without considering the whole path does not guarantee that the full monitoring procedure has the intended error rate.

Continuous monitoring also has practical qualifications. A real system observes data at finite resolution, so its implemented rule may in fact check at discrete times. The timing and frequency of those checks should be stated. When there is a gap between checks, a crossing could occur and disappear between them. In a model with continuous paths, a sufficiently fine grid can approximate the maximum, but a grid calculation is still an approximation unless its error is assessed. If the data-generating process, variability, or boundary differs from the assumed model, the simulated or calculated crossing probability may not describe actual performance.