The Tradeoff Between Monitoring and False Positives

Why Monitoring Creates a Tradeoff

A planned interim analysis can matter because a trial’s result may become useful before its maximum sample size is reached. If evidence of benefit is persuasive, an early decision may allow an effective treatment to be offered sooner or avoid enrolling additional participants who may receive a less effective option. If the evidence suggests futility or harm, an early decision may also prevent unnecessary exposure and resource use.

But each opportunity to declare success is also another opportunity to reject \(H_0\) when it is true. A test that uses significance level \(\alpha=0.05\) controls the false-positive probability for that one test under its assumptions. It does not, by itself, guarantee a 5% false-positive probability across several tests conducted over time. Monitoring therefore involves a real design tradeoff: the value of timely information must be weighed against the risk of a false claim, and repeated testing needs an explicit error-control plan.

The probability is evaluated under the trial’s null model and the specified monitoring and stopping rules. When the null includes several parameter values, control is usually required over the null values relevant to the claim. The discussion here assumes a well-defined primary outcome, a prespecified null and alternative, valid test statistics at each analysis, and a fixed schedule of planned looks. It also assumes that the trial’s decisions and analyses follow that plan. These assumptions make it meaningful to describe the repeated-testing risk; they do not ensure that a chosen plan controls it.

Why More Unadjusted Looks Can Mean More False Positives

Let \(A_k\) be the event that the test rejects \(H_0\) at look \(k\). A false positive at some point in the trial occurs if at least one \(A_k\) occurs. Thus the overall error probability is the probability of the union of the rejection events:

The probability of a union cannot be smaller than the probability of any one event it contains. Adding a look while keeping all previous rejection rules unchanged therefore cannot reduce this probability. How much it increases depends on how the test results at different looks are related. In a cumulative trial, looks usually reuse data, so their results are dependent. The independent-test calculation below is a benchmark for understanding accumulation, not an exact calculation for such a trial.

This union bound follows because the indicator of a union is no greater than the sum of the indicators of its component events; taking expectations gives the probability inequality. If each test has false-positive probability \(0.05\), the bound is \(0.05K\), capped at 1. It is a bound, not generally the exact overall error. Dependence and overlap between rejection events determine the actual probability.

A key distinction follows. Adding an unadjusted look with the same rejection rule cannot reduce the chance of a false positive, but adding a look as part of a redesigned, error-controlled plan does not necessarily increase the overall error. The rejection rules can be chosen together so that the overall probability is controlled at the desired level. Such control is what permits meaningful comparisons between plans with different numbers or timings of looks.

What the Benefit of Early Monitoring Looks Like

Early monitoring is most valuable when a result can lead to a timely, justified action. A statistically significant interim result that is not reliable enough to support a clinical or operational decision does not deliver the full benefit of early monitoring. The practical value depends on factors such as how quickly participants accrue, how urgent the decision is, the consequences of a delay, and whether stopping early would leave important questions unanswered.

The following example uses hypothetical operating characteristics: probabilities calculated for a specified plan under a particular alternative. They are not universal properties of having three looks. The stopping probabilities sum to 1 because, under this alternative, each trial either stops for efficacy at one of the first two looks or continues to the final look.

Under an alternative, a plan can also be summarized by the chance of rejecting at each look, the chance of stopping early, and the distribution of time to a decision. Those quantities help describe how quickly a plan may detect an effect. They must be considered alongside the overall type I error, since a plan that declares success more readily may appear faster precisely because it tolerates more false positives.

Comparing Plans on Equal Error-Control Terms

A fair comparison does not contrast a carefully controlled monitoring plan with a fixed-horizon test while ignoring their different false-positive risks. Instead, compare plans that control overall type I error at the same target and examine what each offers under plausible alternatives and operational conditions. Relevant summaries include power under specified alternatives, expected sample size, probability of an early decision, and time to decision. No single summary captures every benefit or cost.

A single final analysis is the simplest alternative: it avoids repeated testing over time but cannot deliver an efficacy decision before the planned final analysis. At the other extreme, repeatedly applying an ordinary fixed-sample cutoff without accounting for the looks may offer apparent speed but can inflate the overall false-positive probability. A prespecified, error-controlled monitoring plan offers a middle ground: it allows interim decisions while defining rejection rules for the sequence as a whole.

Assumptions, Diagnostics, and Limitations

A monitoring plan is only interpretable if the analysis schedule and decision rules are specified clearly enough to evaluate. At a minimum, the plan should identify the outcome, the null and alternative, the number and timing of looks, the criterion for stopping, and the method used to assess overall error. The planned analysis should also use valid data and test procedures at each look. If the schedule or decision rules change, the operating characteristics of the original plan may no longer describe the trial that was conducted.

Useful checks include confirming that each planned look corresponds to the stated enrollment or information point, that decisions are based on the prespecified outcome and analysis, and that the reported error rate refers to the whole monitoring plan rather than only the final test. Compare observed conduct with the protocol: unplanned analyses, additional outcomes, or undocumented decision changes can create extra opportunities for a positive claim. An analysis that reports a final p-value alone does not show whether earlier looks occurred or whether they affected stopping decisions.

Several limitations matter when weighing the tradeoff. An early estimate can be imprecise, and stopping when results look unusually favorable can leave the observed effect larger than its long-run value. A trial that stops early may also provide less information about safety, variation in treatment response, or outcomes that require longer follow-up. Conversely, continuing after strong evidence of harm may expose participants unnecessarily. These are reasons to consider the purpose and consequences of monitoring, not reasons to ignore error control.

• Do not equate more looks with better monitoring. More frequent review is useful only if it supports an actionable decision and is incorporated into an error-controlled plan.
• Do not treat a per-look cutoff as an overall guarantee. A 0.05 cutoff at each look does not establish 0.05 overall error.
• Do not use the independent-look formula for cumulative data. Shared observations make interim results dependent; use the plan’s appropriate joint error calculation.
• Do not assess a plan only under \(H_0\). Error control is essential, but usefulness also depends on power, timing, expected sample size, and the consequences of stopping.