Repeated P-Value Peeking Pitfalls

Why Repeated P-Value Peeking Is a Different Procedure

A p-value below a chosen cutoff is often treated as a signal to stop collecting data and declare a result. But if the same cutoff is checked repeatedly as data accumulate, the procedure is no longer a single test at that cutoff. Each look creates another opportunity to cross the boundary. A sequence of individually reasonable calculations can therefore produce a decision rule with a higher overall false-positive probability than the nominal cutoff suggests.

The issue is not that every p-value computed at an interim look is automatically invalid. Under the assumptions of a correctly specified one-look test, its p-value has the usual null calibration at that look. The problem is using those p-values as repeated chances to reject \(H_0\), while interpreting the final decision as though only one test had been performed.

For this tutorial, consider two-sided tests of \(H_0:\mu=0\) using independent observations from a normal population with known standard deviation \(\sigma=1\). At sample size \(n\), the statistic is \(Z_n=\sqrt{n}\bar{x}\), and the two-sided p-value is \(2[1-\Phi(|Z_n|)]\), where \(\Phi\) is the standard normal cumulative distribution function. Under \(H_0\), the observations are independent and normally distributed with mean zero and variance one, so each fixed-look \(Z_n\) has the standard normal distribution. These assumptions make the example transparent; other analyses require their own suitable test and assumptions.

If a one-look test has significance level \(\alpha=0.05\), its null rejection probability is at most 0.05; it is exactly 0.05 when the test is exact at that level. That statement applies to the test at a specified look. It does not, by itself, guarantee that a procedure which checks at several looks and stops at the first crossing has overall error probability at most 0.05.

A Cumulative-Look Example

Suppose observations arrive in blocks of ten. The analyst checks the two-sided p-value after \(n=10\), \(n=20\), and \(n=30\), stopping if any p-value is below 0.05. The observations below are a clearly hypothetical sequence of simulated standard-normal outcomes under \(H_0\). To keep the arithmetic readable, the block sums are reported to four decimal places; the corresponding cumulative sums give the displayed test statistics.

The statistics at the looks are not independent. For \(m\le n\), the first \(m\) observations contribute to both \(Z_m\) and \(Z_n\). Under the independent, unit-variance normal model, their correlation is

For example, the correlation between \(Z_{10}\) and \(Z_{20}\) is \(\sqrt{10/20}\approx0.7071\). A crossing at one look therefore provides information about the statistic at nearby cumulative looks. It would be incorrect to treat the p-values as independent opportunities when calculating the overall error rate. The earlier material on dependence between cumulative interim tests explains why this shared data matters.

What the Simulation Diagnoses

A null simulation makes the repeated decision rule visible across many hypothetical studies. In each replicate, generate observations under \(H_0\), calculate the p-values at the planned looks, and record whether any p-value falls below the fixed cutoff. The fraction of replicates with at least one rejection estimates the procedure’s overall Type I error probability under the simulated model.

A concise way to state the repeated decision is: reject if at least one look rejects. For three looks, the overall error event is the union of the three look-specific rejection events. Its probability depends on how those events overlap; it is not found by simply treating the p-values as independent. The union bound gives an upper bound, but a simulation under the specified joint data-generating process can estimate the probability directly.

where \(R_j\) denotes rejection at look \(j\). This expression identifies the relevant error event. The key diagnostic is to count a simulated replicate as a rejection if the rule would have rejected at any permitted look, not merely if the last p-value is significant.

How to Diagnose and Avoid the Pitfall

When reviewing a report or planning an analysis, ask what the actual decision rule was. Did the investigators choose one sample size and test once, or did they inspect interim results and stop when a threshold was crossed? A statement such as “the p-value was 0.04” is incomplete for evaluating the repeated procedure unless the number and timing of earlier looks, and the stopping rule, are also known.

• Count the looks. Include each analysis that could have triggered stopping or changed the analysis plan, not just the final analysis that appears in a report.
• Specify the rejection event. “Any p-value below 0.05” differs from “the p-value at the one prespecified final look is below 0.05.”
• Check the null calibration of the whole rule. A null simulation can help when the model and monitoring schedule are clear; its replicate-level outcome must be rejection at any look.
• Do not assume independent p-values. Cumulative tests share observations, so the look-specific statistics are dependent.
• Keep the scope of the conclusion clear. A single observed p-value does not reveal the overall error probability of a procedure whose earlier looks and stopping rule are unknown.

The main alternatives are not interchangeable. A single prespecified analysis avoids repeated testing by making the formal decision at one planned look, though it may not suit every monitoring need. A sequentially calibrated design plans the looks and uses decision boundaries chosen to control the overall error rate. Simply continuing to use 0.05 at every look is not a substitute for that calibration. Informal, unplanned peeking may be useful for operational awareness, but treating its first crossing as a confirmatory result requires accounting for the monitoring process.

Common Mistakes and Limits

The most common mistake is to reason, “Each test uses \(\alpha=0.05\), so the chance of a false positive is 5%.” That conclusion applies to a single test at a fixed look, not automatically to the event of rejecting at least once across repeated looks. The correction is to define the full decision rule and evaluate its overall null rejection probability.

Another mistake is to claim that every interim p-value is “wrong” after peeking. The calculations may still be valid p-values for their respective fixed-look tests under the model. What fails is the unadjusted interpretation of the data-dependent stopping decision as if it had been a single test. Conversely, observing that p-values are dependent does not make repeated peeking harmless: dependence changes the amount of inflation, but does not establish that the overall error remains at the per-look level.

A simple simulation is informative only for the conditions it represents. If the outcome distribution, variance, look timing, missing-data process, or test statistic differs materially from the simulation, its estimated error rate may not apply. A finite simulation also has Monte Carlo uncertainty. It is a diagnostic for a specified procedure, not a universal correction or a guarantee for an unmodeled design.