Optional Stopping and False-Positive Claims

Why the Stopping Decision Matters

A significance test is calibrated for a specified analysis procedure. In a fixed-sample test, that procedure includes the sample size and the rule for rejecting \(H_0\). If the test has significance level \(\alpha\), then, under \(H_0\), its probability of rejection is at most \(\alpha\), assuming the model and test conditions hold. The guarantee concerns the procedure as a whole, not just the final arithmetic that produces a p-value.

The issue changes when researchers examine results during data collection and decide whether to stop partly because of what those results show. A nominally significant p-value from one look still describes that look's test under its fixed-sample assumptions. It does not, by itself, describe the probability that the full stopping-and-testing procedure would produce a false-positive claim. That overall probability depends on the rule for looking, stopping, and claiming significance.

The key distinction is between a fixed-look guarantee and a procedure-wide guarantee. If a test would have a valid p-value at each prespecified sample size, that fact alone does not imply that the p-value reported at a data-selected sample size is valid at the same level. The sample size, the decision to stop, and the decision to make a claim can be statistically linked.

Fixed-Sample Validity and a Data-Selected Look

Let \(p_j\) denote the p-value at look \(j\), and suppose that, at any one fixed look, the test is valid under \(H_0\): for every \(u\) between 0 and 1, \(P(p_j \leq u \mid H_0) \leq u\). This inequality is the defining calibration property of a valid p-value. It ensures that a single test rejecting when \(p_j \leq \alpha\) has Type I error at most \(\alpha\).

Now let \(\tau\) be the look selected by a stopping rule, so the reported p-value is \(p_\tau\). If \(\tau\) depends on interim results, the event \(p_\tau \leq \alpha\) is not generally the same as the event that a p-value from one fixed analysis is at most \(\alpha\). For example, a rule that stops and claims success at the first \(p_j \leq \alpha\) rejects whenever at least one look crosses the threshold. Its null rejection event is

if the procedure continues whenever the threshold has not been crossed and makes no other rejection. The probability of this union is not generally bounded by \(\alpha\). Its value depends on the number of looks, their dependence, and the full stopping and rejection rules. Earlier material on repeated tests establishes why repeated opportunities to cross a threshold can change the overall error rate; the point here is that an outcome-dependent stopping decision is part of that same inferential procedure.

Not Every Stopping Rule Has the Same Effect

Optional stopping does not mean that every change in sample size necessarily increases the Type I error rate. The direction and size of any change depend on the stopping rule and test statistics. Stopping because a result looks significant creates a different selection mechanism from stopping because a study reaches a calendar deadline, and either differs from a rule that stops for futility and suppresses a claim. The correct question is not simply whether stopping occurred, but whether the procedure's null rejection probability is still controlled.

Why the Nominal P-Value Can Mislead

Suppose a report says, “The final p-value was 0.04, so the study had a 5% false-positive rate.” That conclusion is justified only if the reported p-value comes from an analysis procedure for which the 5% guarantee holds. If the study repeatedly examined results and stopped when evidence seemed sufficient, the fixed-sample p-value at the stopping point does not account for the selection of that point. A p-value of 0.04 is not a 4% probability that \(H_0\) is true, nor does it directly give the chance that the study's overall claim is false.

The p-value can still be calculated correctly for the model and sample size used at the final look. The problem is not necessarily an arithmetic error. It is an interpretation error when that number is treated as if the final look had been selected in advance, although the decision to reach or report it was influenced by earlier results. The relevant error rate is the probability, under \(H_0\), that the entire procedure would make a claim.

Practical Checks and Alternatives

Before interpreting a reported significance result, establish how the analysis was selected. Useful checks include whether the sample size and analysis time were set independently of observed outcomes, whether interim results were examined, whether stopping could be triggered by apparent benefit, and whether all analyses capable of supporting the claim were included in the stated procedure. If the rule is unclear, a reader cannot verify that the nominal level describes the overall false-positive risk.

There are several broad approaches, with different tradeoffs. A genuinely fixed-sample analysis uses one planned analysis and its single-test calibration. Repeated unadjusted tests apply the same nominal cutoff at multiple looks but do not generally preserve that single-test error rate for an any-look claim. A calibrated sequential procedure specifies how evidence will be assessed across monitoring and controls the procedure-wide error rate under its assumptions. The appropriate choice depends on the scientific and operational aims; one should not relabel an unadjusted repeated-testing result as a fixed-sample result.

A common mistake is to argue, “The final analysis used a valid 0.05 test, so the overall chance of a false positive is at most 5%.” This confuses validity at one fixed analysis with validity after selecting an analysis based on observed data. The correction is to define the complete procedure, including when results are examined, how stopping occurs, and what counts as a rejection, and then assess its Type I error rate under \(H_0\).

This lesson has limits. It does not say that every nominally significant result after stopping is false, that every data-dependent rule inflates error, or that a p-value is unusable whenever a study changes course. It says that the usual single-test guarantee cannot be assumed without checking the procedure. Correct calibration also depends on the statistical model and other test assumptions; accounting for stopping cannot repair unrelated model violations.