A Worked Two-Look Error Example

Why a Two-Look Calculation Is Useful

If a study makes two chances to reject a true null hypothesis, the chance of a false rejection across the study is generally larger than the error rate of either test alone. The size of that increase depends on how the two rejection events relate. This tutorial isolates a deliberately simple case: two independent tests, each with a known type I error rate. It provides a clean numerical benchmark and makes clear which assumptions are needed before applying the calculation.

The word “look” can describe a scheduled analysis, but the calculation below does not assume that the same accumulating data are tested twice. Instead, imagine two tests whose rejection outcomes are independent under the null. Independent samples or separately generated test statistics may support that assumption. In a typical interim analysis of cumulative data, the tests share observations, so their rejection events are not automatically independent; the earlier material on dependence between repeated tests addresses that distinction.

Definition: Let \(R_1\) and \(R_2\) denote the events that tests 1 and 2 reject their null hypotheses. When the relevant null hypotheses are true, the any-look false-rejection probability is \(P(R_1\cup R_2)\): the probability of at least one false rejection across the two tests. If both tests concern the same true null, and each has type I error rate \(\alpha\), then \(P(R_1)=P(R_2)=\alpha\). The exact calculation below additionally requires \(R_1\) and \(R_2\) to be independent under that null.

Deriving the Two-Look Probability

It is often simplest to calculate the probability of the complementary event: neither test rejects. Under independence, the probability of no rejection at either look is the product of the two probabilities of no rejection. Subtracting this from 1 gives the probability of at least one rejection.

P(R_1\cup R_2) =1-P(R_1^c\cap R_2^c) =1-P(R_1^c)P(R_2^c) =1-(1-\alpha)^2.

Here \(R_i^c\) means that test \(i\) does not reject. The formula concerns the probability of the union, not the probability that both tests reject. For example, at \(\alpha=0.05\), the probability of two rejections is \(0.05^2=0.0025\), whereas the probability of at least one rejection is much larger because it includes outcomes in which only the first or only the second test rejects.

The same result follows by separating the paths under a first-rejection rule. A false rejection can happen at the first test, or the first test can not reject and the second can then reject. Independence makes the second test’s rejection probability \(\alpha\) on the path where the first did not reject.

P(\text{at least one rejection}) =P(R_1)+P(R_1^c)P(R_2\mid R_1^c) =\alpha+(1-\alpha)\alpha =1-(1-\alpha)^2.

This path calculation also shows why merely adding the two nominal error rates overstates the exact answer when the events are independent: some probability is counted twice, namely the probability that both tests reject. The union bound gives \(P(R_1\cup R_2)\leq P(R_1)+P(R_2)=2\alpha\); independence gives the more precise value \(2\alpha-\alpha^2\).

Worked Example: Two Independent Tests at 5%

Suppose a research team plans two tests of the same treatment-effect null hypothesis, each at a two-sided significance level of 0.05. For this hypothetical illustration, the tests use separate independent samples, and each test is correctly calibrated to have a 0.05 rejection probability when the null is true. A possible observed pair of p-values is 0.031 and 0.42: the first test rejects, and the second does not. The p-values describe that particular outcome; the any-look probability is calculated from the design assumptions, not from those observed p-values.

Under the true null, each test has nonrejection probability \(1-0.05=0.95\). Independence gives:

P(\text{at least one false rejection}) =1-(0.95)(0.95) =1-0.9025 =0.0975.

Thus, if the null is true and the independent-test assumptions hold, the chance of at least one false rejection is 9.75%. A single test at the same nominal level has false-rejection probability 5%. The two-test any-look probability is 4.75 percentage points higher, or 1.95 times the single-test rate. It is not correct to call it 10% exactly: \(0.05+0.05=0.10\) is the union-bound upper limit, while independence subtracts the overlap \(0.05^2=0.0025\).

Different Rates and a Chosen Overall Target

The tests do not have to use identical significance levels. If test 1 has null rejection probability \(\alpha_1\) and test 2 has null rejection probability \(\alpha_2\), independence gives the probability of no rejection as \((1-\alpha_1)(1-\alpha_2)\). The any-look probability is therefore one minus that product. This calculation requires both rates to apply under the null configuration being considered.

Worked Example: Unequal Test Levels

Imagine two independent hypothetical studies testing the same null hypothesis, with test 1 using a 0.02 significance level and test 2 using a 0.05 significance level. Assume both tests are valid and independent under the true null. Suppose their observed p-values are 0.027 and 0.044; neither is below its own threshold, so neither rejects in this particular realization. The planned any-look error rate is a separate probability calculation:

P(\text{at least one false rejection}) =1-(1-0.02)(1-0.05) =1-(0.98)(0.95) =1-0.931 =0.069.

The any-look false-rejection probability is 6.9%, compared with 2% for test 1 alone and 5% for test 2 alone. The observed p-values do not alter those design-level rates. They simply show that neither rejection occurred in the hypothetical realization. Adding the levels gives the upper bound \(0.02+0.05=0.07\); the exact independent-test value is 0.069 because the probability of both rejections, \(0.02(0.05)=0.001\), was counted twice in the sum.

The formula can also be reversed. If the goal is a specified overall error probability and both independent tests will use the same per-test level, solve for that level rather than treating it as 0.05 by default. This is a basic way to see how a per-test threshold must change when more than one independent chance to reject is planned.

Worked Example: Choosing Equal Levels for a 5% Overall Rate

Suppose investigators want the chance of at least one false rejection across two independent tests to be 0.05. Let \(\alpha_*\) be the common per-test significance level. Under the true null, the probability of no rejection at either test must be 0.95:

1-(1-\alpha_*)^2=0.05.

Rearrange and take the nonnegative square root:

(1-\alpha_*)^2=0.95,\qquad 1-\alpha_*=\sqrt{0.95},\qquad \alpha_*=1-\sqrt{0.95}\approx 0.02532.

Each independent test would therefore use a level of about 2.532% to make the overall any-look rate 5%. Checking by substitution:

1-(1-0.02532)^2 \approx 1-(0.97468)^2 \approx 0.0500.

This is an exact allocation for two equally sized, independent tests under the stated assumptions. It is close to, but not identical to, assigning 0.025 to each test. That latter allocation yields \(1-(0.975)^2=0.049375\), or 4.938% overall.

What Independence Does—and Does Not—Buy

For two tests with potentially different rejection probabilities, the general union identity is:

P(R_1\cup R_2) =P(R_1)+P(R_2)-P(R_1\cap R_2).

The independence assumption supplies \(P(R_1\cap R_2)=P(R_1)P(R_2)\). If each test has rate \(\alpha\), this gives \(2\alpha-\alpha^2\), the same value as \(1-(1-\alpha)^2\). Without independence, the marginal rates alone do not determine the exact probability: the overlap of the rejection events is also needed.

For perspective, two identical rejection events would have an any-look probability of \(\alpha\), not \(2\alpha-\alpha^2\). That limiting case illustrates why the independence formula cannot simply be transferred to repeated tests of shared, cumulative data. In such analyses, outcomes at different looks can be dependent. The schedule and the data structure matter; a formula based only on the number of looks and the per-look rate is not enough unless its dependence assumptions are justified.

The union bound remains valid without independence: if both marginal rates are \(\alpha\), then \(P(R_1\cup R_2)\leq 2\alpha\). It is a bound, not generally an exact answer. The independent calculation is more specific and should be used only when independence of the rejection events under the null is defensible. For cumulative interim tests, the appropriate joint behavior must be accounted for rather than assumed away.

Assumption Checks and Common Pitfalls

Before reporting a two-look error probability, check each part of the calculation’s setup:

Null configuration: The probability being called a false-rejection probability is evaluated when the relevant null hypothesis or hypotheses are true. If only some hypotheses are true, the error event and its interpretation need to be stated accordingly.
Valid per-test levels: Each test must have the claimed rejection probability under its null. A nominal threshold does not guarantee that rate if the test is miscalibrated or its assumptions fail.
Independence: The rejection indicators must be independent under the null. Separate independent samples can make this plausible; testing overlapping data does not establish it.
Planned testing rule: The calculation assumes the stated two chances to reject are the relevant opportunities. Additional analyses or unplanned looks introduce further rejection opportunities not included in the formula.

A common mistake is to say, “Each look is at 5%, so the study’s false-positive rate is 5%.” That treats the rate for one test as if it were the rate for the event “at least one of two tests rejects.” Under independent tests at 5%, the correction is \(1-(0.95)^2=0.0975\), or 9.75%. A second mistake is to say the exact rate must be 10% by adding the two rates. Addition gives the union bound; under independence, subtract the overlap \(0.0025\), obtaining 9.75%.

Key takeaway: For two independent tests with true-null rejection probabilities \(\alpha_1\) and \(\alpha_2\), the exact chance of at least one false rejection is \(1-(1-\alpha_1)(1-\alpha_2)\). With two tests each at 5%, this is 9.75%, not 5%. The calculation is not automatically valid for cumulative interim tests whose rejection events are dependent.

Check Your Understanding

Answer each question, then compare with the solutions.

Two independent tests each have a true-null rejection probability of 0.04. What is the probability of at least one false rejection?
Two independent tests have null rejection probabilities 0.01 and 0.03. Calculate the any-look probability and compare it with the union bound.
Why is it not enough to know that two tests each have a 5% level to conclude that their any-look error is exactly 9.75%?
For two independent tests with the same level, find the per-test level that gives an any-look probability of 0.10.
Explain why the 9.75% calculation should not automatically be used for two tests on cumulative data.

Solutions. (1) \(1-(1-0.04)^2=1-(0.96)^2=0.0784\), so the probability is 7.84%. (2) \(1-(0.99)(0.97)=1-0.9603=0.0397\), or 3.97%. The union bound is \(0.01+0.03=0.04\), or 4%; it is slightly larger. (3) The exact 9.75% value also requires independence of the rejection events, in addition to valid 5% rates under the null. (4) Solve \(1-(1-\alpha_*)^2=0.10\), giving \(\alpha_*=1-\sqrt{0.90}\approx0.05132\), or 5.132% per test. (5) Cumulative tests share data, so their rejection events may be dependent. The marginal 5% rates alone do not determine their joint probability; using the independent formula without justification can give the wrong any-look error.