Two Different Error Probabilities
In a study with repeated significance tests, “the error rate” can refer to different events. A test may reject at the planned final analysis, or it may reject at any scheduled analysis, including an interim one. These probabilities are not generally the same. Distinguishing them matters because a study can produce a false positive before its final analysis, even if its final-look result would not be significant.
Consider a fixed schedule of \(K\) looks and a null hypothesis \(H_0\). Imagine the data and test statistics at all scheduled looks as defined on one probability space under \(H_0\). This common setup lets us compare the events directly. If a study stops after an interim rejection, the later statistics may not actually be observed; the final-look event still has a clear meaning as the event that would occur at the planned final analysis under the same data-generating process.
The final-look error probability concerns one specified test at the planned final analysis. The any-look error probability concerns whether at least one rejection occurs over the whole monitoring schedule. When each look uses a nominal significance level \(\alpha\), the final-look probability may be \(\alpha\) under \(H_0\), but this alone does not imply that the any-look probability is \(\alpha\).
The Basic Relationship
The final look is one of the scheduled looks, so every final-look rejection is also an any-look rejection. Thus \(R_K\subseteq R_{\mathrm{any}}\). The any-look event can be partitioned into final-look rejections and any-look rejections that did not occur at the final look:
The two events in the union are disjoint, which justifies adding their probabilities. In particular, the any-look error probability cannot be smaller than the final-look error probability. The size of the difference is exactly the probability, under \(H_0\), of a rejection path that includes at least one rejection but no final-look rejection.
This condition is more precise than saying that final rejections are “covered” by earlier rejections. It is not enough for every final rejection to have an earlier rejection as well: earlier looks may also reject on paths where the final test does not. Those extra paths increase the any-look probability.
Worked Example: Two Independent 5% Tests
Suppose two planned tests are conducted under \(H_0\). Each has rejection probability \(0.05\), and the two rejection events are independent. Let \(R_1\) and \(R_2\) denote rejection at the first and final looks, respectively. The final-look error is \(P(R_2\mid H_0)=0.05\).
Independence gives \(P(R_1^c\cap R_2^c\mid H_0)=0.95\times0.95=0.9025\), where \(R_k^c\) means no rejection at look \(k\). Therefore,
The any-look error is \(9.75\%\), while the final-look error is \(5\%\). The difference, \(0.0975-0.05=0.0475\), is the probability of rejecting at the first look but not at the final look. The example assumes independent rejection events; that assumption is useful for this calculation but is not generally appropriate for cumulative interim analyses.
Why Dependence Changes the Comparison
In a cumulative study, successive test statistics are usually calculated from overlapping data. Their rejection events are therefore dependent. The union identity remains valid, but calculating the union requires accounting for the overlap between rejection events. For two looks,
The intersection term prevents double-counting paths that reject at both looks. If the marginal rejection probabilities are fixed, a larger intersection generally means fewer distinct paths contribute to the union. Dependence does not change the marginal error probability at a look when its test statistic still has the specified null distribution; it changes how the rejection events overlap.
Worked Example: Correlated Two-Sided Normal Tests
Suppose, hypothetically, that under \(H_0\), the two look statistics \(Z_1\) and \(Z_2\) have a bivariate standard normal distribution with correlation \(0.5\). Each look rejects when \(|Z_k|>1.959964\), the two-sided standard normal cutoff giving a marginal rejection probability of \(0.05\). Thus the final-look error is \(0.05\). The bivariate normal model and the stated correlation are assumptions for this example, not a claim that every monitoring design has this joint distribution.
For \(c=1.959964\), write \(Q(c)=P(Z>c)\) for a standard normal variable \(Z\), and let \(\phi_2(x,y;r)\) be the bivariate standard normal density with correlation \(r\). The same-direction upper-tail probability can be written as
For the opposite-direction upper-and-lower quadrant, the corresponding expression is \(Q(c)^2-\int_0^{0.5}\phi_2(c,-c;r)\,dr\). Adding the two same-direction quadrants and the two opposite-direction quadrants gives, by numerical evaluation, \(P(|Z_1|>c,\ |Z_2|>c\mid H_0)\approx0.00926\). In particular, the upper-upper quadrant is about \(0.00462\); the two-sided intersection includes all four quadrants, not just the two upper tails.
Using the union formula and the two marginal probabilities,
The any-look error is therefore about \(9.07\%\), compared with a final-look error of \(5\%\). The gap is about \(0.0407\), the probability of an any-look rejection without a final-look rejection. The correlated result is below the \(9.75\%\) independent result because, here, the two tests reject together more often. It is still greater than \(5\%\), because the earlier look adds rejection paths.
Equality, Added Paths, and a Common Mistake
Worked Example: A Final Rejection Does Not Guarantee Equality
Consider two rejection events under \(H_0\) with the following hypothetical probabilities: \(P(R_1\mid H_0)=0.08\), \(P(R_2\mid H_0)=0.05\), and \(R_2\subseteq R_1\). Here \(R_2\) is the final-look event. Because every final rejection is also a first-look rejection, \(R_{\mathrm{any}}=R_1\), so \(P(R_{\mathrm{any}}\mid H_0)=0.08\), not \(0.05\).
In this example, \(P(R_{\mathrm{any}}\setminus R_2\mid H_0)=0.08-0.05=0.03\). Thus the earlier look adds rejection paths even though every final rejection is accompanied by an earlier rejection. Equality would require the extra-path probability to be zero, not merely that final rejections occur together with earlier ones.
Common mistake: “The final test has a 5% significance level, so the repeated procedure has a 5% chance of a false positive.” This reasoning substitutes the probability of one event, \(R_K\), for the probability of the union of all rejection events. The correction is to name the event being reported and calculate its probability. A final-look probability can be \(5\%\) while the probability of at least one false rejection is larger.
A second mistake is to use independence automatically. Cumulative tests often share observations, so their test statistics can be strongly dependent. The independent formula \(1-(1-\alpha)^K\) applies only when the rejection events are independent and each has probability \(\alpha\). For dependent looks, use the joint distribution or an appropriate null simulation to estimate the any-look probability; do not infer it from the marginal probabilities alone.
What to Compare and What to Report
The final-look and any-look probabilities answer different questions. The final-look error describes the chance of rejection by the test performed at the planned final analysis. The any-look error describes the chance that the monitoring rule rejects at least once over all scheduled looks. If the goal is to characterize false-positive risk for the whole monitoring procedure, the any-look probability is the relevant quantity. A per-look significance level is not, by itself, a statement of that overall risk.
| Quantity | Event under \(H_0\) | What it describes |
|---|---|---|
| Final-look error | \(R_K\) | Rejection at the planned final test |
| Any-look error | \(\bigcup_{k=1}^K R_k\) | At least one rejection over the monitoring schedule |
| Added-path probability | \(R_{\mathrm{any}}\setminus R_K\) | Rejection somewhere, but not at the final look |
The main alternative to an any-look calculation is to report only the final-look probability. That is appropriate only when the question is explicitly about the final test in isolation; it does not summarize the chance of a rejection anywhere in a repeated-testing procedure. In a design intended to control error over the entire schedule, the rejection rule must be evaluated as a whole. The comparison here identifies why that distinction matters, without assuming that the final-look test alone represents the full procedure.
For diagnosis, check that the event used in a calculation matches the claim being made, that the number and timing of looks are specified, and that the dependence assumptions are credible. In particular, compare marginal rejection probabilities with the joint rejection probability: if the overlap is omitted, the union is miscalculated. Also check whether the reported analysis truly uses the full planned schedule. If testing can stop early, distinguish the actual stopping rule from a hypothetical final-look-only analysis.
These probabilities describe behavior under \(H_0\); they do not determine the probability of rejection under an alternative hypothesis. They also depend on the monitoring schedule, the test statistics, their joint distribution, and the rejection rule. A numerical any-look error from one schedule should not be transferred to a different schedule without checking those features.
Decide whether the claim concerns rejection at the final look or at least one rejection across all looks.
Specify the marginal null distributions and, when needed, the dependence between looks.
For an any-look event, account for overlap among the rejection events rather than treating marginal probabilities as disjoint.
The excess of any-look over final-look error is exactly the probability of any-look rejection without final-look rejection.
Check Your Understanding
Answer each question, then check the solutions.
- Why must the any-look rejection probability be at least as large as the final-look rejection probability?
- Two independent looks each have a null rejection probability of \(0.02\). What are the any-look and final-look error probabilities?
- Suppose two looks each have marginal rejection probability \(0.05\), and their joint rejection probability is \(0.01\). Find the any-look probability.
- Is equality of the any-look and final-look probabilities guaranteed if every final-look rejection is also an earlier-look rejection? Explain.
- Why is the independent-look formula not automatically appropriate for cumulative interim tests?
Solutions. 1. Since \(R_K\subseteq R_{\mathrm{any}}\), the any-look event contains the final-look event. 2. Independence gives \(1-(1-0.02)^2=0.0396\), or \(3.96\%\), for any-look error; the final-look error is \(2\%\). 3. By inclusion-exclusion, \(0.05+0.05-0.01=0.09\), or \(9\%\). 4. No. Earlier looks can add rejection paths that do not reject at the final look; equality requires \(P(R_{\mathrm{any}}\setminus R_K\mid H_0)=0\). 5. Cumulative tests commonly use overlapping data, making their rejection events dependent; the independent formula requires independence.