Common Errors in Reporting Repeated Tests

Why Reporting Repeated Tests Requires Precision

A report that says “the test used a 5% significance level” can leave an important question unanswered: was 5% the threshold at each analysis, or was it the error rate controlled across all opportunities to reject? When results are examined repeatedly, these are generally not the same claim. A reader needs to know how many chances there were to reject, what rule governed those chances, and whether the study stopped after an interim result.

The issue is not merely terminology. A per-look threshold describes a decision at one analysis. Trial-wise type I error describes the probability, under the null hypothesis, of at least one false rejection across the entire monitoring process. Reporting the first as though it were the second can make a procedure sound more protective than it is. Omitting interim rejection opportunities can make the analysis sound like a single final test when it was not.

Definition: Let \(K\) be the number of planned looks, and let \(p_j\) be the p-value at look \(j\). A per-look threshold \(\alpha\) is a rule to reject \(H_0\) at a particular look when \(p_j \leq \alpha\). Under a specified monitoring and stopping rule, the trial-wise type I error is the probability, when \(H_0\) is true, of rejecting \(H_0\) at one or more looks. These quantities answer different questions.

For this distinction to be meaningful, the testing procedure and its assumptions must be clear. Suppose each look uses a test whose p-value is valid under \(H_0\), so that at a fixed look \(P(p_j \leq \alpha \mid H_0) \leq \alpha\). The look schedule and the rule for stopping or rejecting must also be described. The probability of at least one rejection depends on the joint behavior of the tests and on the decision rule; the per-look threshold alone does not determine it. In particular, one must not assume that tests at different looks are independent unless that assumption is justified.

Per-Look Threshold and Trial-Wise Error

Let \(E_j\) denote the event that the test rejects \(H_0\) at look \(j\). If the monitoring rule rejects as soon as any look crosses the threshold, the overall rejection event is \(E_1 \cup E_2 \cup \cdots \cup E_K\). The type I error for that rule is \(P(E_1 \cup \cdots \cup E_K \mid H_0)\), not simply the probability at one chosen look.

A basic probability bound explains why reporting a per-look threshold as if it were a trial-wide guarantee is unsafe. The probability of a union of events is at most the sum of their probabilities. Thus, if each of \(K\) tests has type I error at most \(\alpha\), the probability of any rejection is at most \(K\alpha\), with the bound capped at 1. This bound need not be exact, but it shows that the per-look statements alone do not establish trial-wise error control at \(\alpha\).

P\left(\bigcup_{j=1}^{K} E_j \mid H_0\right) \leq \sum_{j=1}^{K} P(E_j \mid H_0) \leq K\alpha

If, in addition, the \(K\) tests are independent under \(H_0\) and each has exact type I error \(\alpha\), then the probability of no rejection is \((1-\alpha)^K\). The probability of at least one rejection is therefore \(1-(1-\alpha)^K\). Repeated tests based on accumulating or shared data are often dependent, so this formula is not a general repeated-testing formula. It is useful only when its independence and exact-size assumptions hold.

Worked Example: Correcting a Two-Look Error Claim

Hypothetical scenario. A study plans two looks at its data. At each look, it rejects \(H_0\) if the p-value is at most 0.05. For illustration, assume that under \(H_0\) the two tests are independent and each has exact type I error 0.05. A draft report says, “The trial had a 5% chance of a false positive.”

The stated assumptions let us calculate the probability of at least one false rejection. The chance of no rejection at either look is \(0.95 \times 0.95 = 0.9025\). Therefore,

P(E_1 \cup E_2 \mid H_0) = 1 - P(E_1^c \cap E_2^c \mid H_0) = 1 - (0.95)^2 = 0.0975.

Under these assumptions, the trial-wise type I error is 0.0975, or 9.75%, even though the per-look threshold is 0.05. The draft confuses the threshold at each look with the probability of any false rejection across the trial.

A more accurate report is: “At each of two planned looks, the unadjusted per-look significance threshold was 0.05. Under the hypothetical independence and exact-size assumptions, the probability of at least one false rejection under \(H_0\) was 0.0975.” In an actual study, the trial-wise probability should be reported from a valid calculation for that study’s joint testing procedure; independence must not be inserted just to obtain a number.

Describe the Opportunities to Reject

A concise methods description should let a reader reconstruct what could lead to a rejection. At a minimum, state the planned number and timing of looks, which analyses could trigger rejection, the decision threshold or rule at each look, and whether the study stopped when a rejection criterion was met. If the study did not follow the plan, report what actually happened as well as what was planned. A final p-value alone does not reveal whether earlier tests could have stopped the study.

Keep the description of the rule separate from the report of the observed result. The rule says what would count as evidence for rejection at each look; the observed results say which analyses occurred and what they found. Calling a p-value “unadjusted” or “per-look” can help prevent readers from interpreting it as a trial-wise error guarantee. Do not call an unadjusted p-value an “overall p-value” unless it was calculated for the full monitoring procedure.

Worked Example: Reporting the First Crossing and Stopping

Hypothetical scenario. A study has three planned looks, at 50%, 75%, and 100% of its target information. The protocol says to stop and reject \(H_0\) at the first look where the p-value is at most 0.05. The observed p-values are 0.18 at the first look and 0.032 at the second. The study stops at the second look.

The observed result is a rejection at look 2 because \(0.032 \leq 0.05\). The third planned look did not occur. It would be inaccurate to report this as though the study had only one analysis, or to imply that a final-look result was obtained. It would also be inaccurate to say that 0.032 is a trial-wise-adjusted p-value: the stated rule gives a per-look threshold but does not say that the p-value has been adjusted for repeated opportunities.

A transparent report would say: “The protocol specified looks at 50%, 75%, and 100% of target information, with stopping for rejection at the first look having a p-value of 0.05 or less. The observed p-values were 0.18 at the 50% look and 0.032 at the 75% look; the study stopped at the second look after crossing the per-look threshold. No 100% look was conducted.” This reports the decision and the stopping history without claiming trial-wise error control.

For the type I error claim, the report still needs a valid evaluation of the entire rule under \(H_0\). The observed p-values alone do not provide that probability: type I error is a probability over repeated possible data sets under the null, not a label determined by the one observed sequence.

What a Complete Report Should Make Clear

The goal is not to make every result appear complicated. It is to prevent ambiguity about which analysis was performed and what error-rate statement the design supports. A reader should be able to distinguish the following:

Per-look rule: the threshold or decision criterion applied at an individual analysis.
Trial-wise error claim: the probability of at least one false rejection across the monitoring process under \(H_0\), evaluated for the specified procedure.
Observed path: which looks occurred, the results at those looks, and whether the stopping rule was triggered.

These distinctions also help when comparing approaches. A single prespecified final analysis has one planned rejection opportunity. A repeated-testing procedure has interim opportunities as well as any final one, and its trial-wise error claim must refer to that complete procedure. A procedure designed to control error across repeated looks can be reported as such, but the report must identify the procedure and its trial-wise target rather than relying on the phrase “5% significance level.” The details of how to construct or calibrate such a procedure are separate from the reporting task.

Worked Example: Four Looks and an Error-Rate Bound

Hypothetical scenario. A report describes four scheduled tests, each with a valid per-look type I error no greater than 0.05 under \(H_0\), and says, “The 5% level was maintained throughout the trial.” No further information about dependence or calibration is given.

The phrase “maintained throughout” can accurately describe the threshold at each look, but it does not establish that the trial-wise type I error is 0.05. Without a joint model, we can give the general union bound:

P\left(\bigcup_{j=1}^{4} E_j \mid H_0\right) \leq \sum_{j=1}^{4} P(E_j \mid H_0) \leq 4(0.05) = 0.20.

This bound says only that the probability of any false rejection is at most 0.20 under the stated per-look validity; it does not say the probability equals 0.20. If the four tests were additionally independent and each had exact size 0.05, the probability would instead be \(1-(0.95)^4 = 0.1855\). Since the report supplies no independence assumption or other joint calculation, neither exact value should be asserted for the study.

A corrected description would state that four tests were conducted with a 0.05 per-look threshold and that the trial-wise type I error was not established by that threshold statement alone. If the study used a method that controls trial-wise error, the report should identify that method and its justified trial-wise error rate. The key correction is to avoid saying that the trial-wide false-positive probability was 5% merely because each look used 0.05.

Diagnostics and Common Reporting Mistakes

Before finalizing a report, check whether the analysis description answers these questions: Could an interim result have led to rejection? How many looks were planned, and how many actually occurred? Was stopping possible before the final look? Does the stated error rate refer to one test or to the entire sequence? If any answer is missing, the description may not support the claim readers are likely to infer.

Mistake: “Each look used 0.05, therefore the trial-wise error was 0.05.” This treats several rejection opportunities as one. Report the per-look threshold separately and give a trial-wise error rate only when it has been evaluated for the full procedure.
Mistake: reporting only the final analysis. If interim tests were capable of triggering rejection or stopping, omitting them hides part of the decision rule. State the interim opportunities and the actual stopping path.
Mistake: assuming the independent-tests formula applies automatically. Repeated looks may use overlapping information. Use an independence calculation only when independence under \(H_0\) is justified; otherwise report a valid result for the actual joint procedure or avoid an unsupported numerical claim.
Mistake: treating an observed p-value as a probability that \(H_0\) is true. A p-value describes how incompatible the data are with \(H_0\), under the test’s assumptions. It is not the probability that the null is true, and a single observed value does not itself give the trial-wise type I error.

Key takeaway: Report the per-look threshold, the full set of rejection opportunities, the stopping rule, and the observed path. State a trial-wise type I error only when it is justified for that complete monitoring procedure.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

A study conducts three tests, each at a per-look threshold of 0.05. Is it justified to state that the probability of any false rejection is 0.05? Why or why not?
Under the additional assumption that four tests are independent under \(H_0\) and each has exact type I error 0.05, calculate the probability of at least one false rejection.
A report gives the final p-value but does not say whether interim results could trigger stopping. What information is missing?
Explain why a valid trial-wise error rate cannot generally be recovered from the observed p-values alone.

Solutions. (1) No. The 0.05 threshold describes each look; the probability of at least one false rejection depends on the joint behavior of the tests and the full decision rule. (2) The probability is \(1-(1-0.05)^4=1-0.8145=0.1855\), or 18.55%. (3) The report should state the planned and actual looks, the interim rejection criteria, and whether stopping occurred or could occur before the final analysis. (4) Trial-wise type I error is a probability over repeated data sets under \(H_0\), whereas the observed p-values describe one realized sequence; evaluating that probability requires the testing procedure and its null distribution across looks.