Deriving Inflation Under Independent Repeated Tests

Why Repeated Tests Need an Overall Probability

A significance level describes the false-rejection probability for a single test under its null hypothesis. When a monitoring plan includes several tests, however, the relevant question may be whether any of them rejects. Even if every test has the same per-test significance level, the chance of at least one false rejection can be larger than that level.

Here we derive that overall probability for a specific setting: a fixed number of independent tests, each conducted at the same level, evaluated when the null hypothesis is true. The key step is to calculate the probability of the complementary event—no test rejects—and subtract it from one.

Assumptions and Notation

Consider \(K\) prespecified tests. For test \(i\), let \(R_i\) denote the event that it rejects. Assume that the null hypothesis is true for every test at the configuration being considered. Each test has rejection probability \(\alpha\) at that configuration, and the rejection events \(R_1,\ldots,R_K\) are mutually independent. Mutual independence means that probabilities of joint outcomes factor into the product of their individual probabilities.

The distinction between a test having size exactly \(\alpha\) and having size at most \(\alpha\) matters. The familiar formula below is an equality when every test has rejection probability exactly \(\alpha\) under the null. If each test only controls its rejection probability at or below \(\alpha\), the formula gives an upper bound under independence, rather than necessarily the exact probability.

The number of tests \(K\) is a positive integer, and \(0\leq\alpha\leq1\). Independence concerns the test outcomes under the null configuration, not merely the fact that the tests have the same nominal significance level. Equal levels do not imply independent outcomes.

Deriving the Any-Rejection Probability

It is often simpler to find the probability that no test rejects. For each test, the probability of no rejection is \(1-\alpha\). By mutual independence, the probability that all \(K\) tests fail to reject is the product of those probabilities. The any-rejection event is the complement of this no-rejection event.

The product step is exactly where independence is used. The final expression is not obtained by simply adding the \(K\) per-test probabilities: the same overall event may be counted more than once by such an addition when multiple tests reject. The complement calculation accounts for all possible patterns of rejections at once.

If independent tests have rejection probabilities \(p_1,\ldots,p_K\), which need not be equal, the same argument gives \(1-\prod_{i=1}^{K}(1-p_i)\). In particular, if each \(p_i\leq\alpha\), then the probability is at most \(1-(1-\alpha)^K\). This follows because each no-rejection probability \(1-p_i\) is at least \(1-\alpha\), so their product is at least \((1-\alpha)^K\).

Worked Calculations

Interpreting the Formula and Checking Its Use

The expression \(1-(1-\alpha)^K\) connects a per-test level to an overall probability for the specific independent-test setup. When \(K=1\), it reduces to \(\alpha\), as it must: with one test, “at least one rejection” is just that test’s rejection. For \(0<\alpha<1\), adding another independent test increases the any-rejection probability, because there is another opportunity to reject.

This probability is defined under the null configuration. It is not the probability that the null hypothesis is true after observing a rejection, nor is it the probability that a particular finding is false. It describes how often the specified collection of tests would produce one or more rejections if the null configuration held and the stated assumptions were satisfied.

Before applying the result, check the following:

• Is the event being counted “at least one rejection”? If the question concerns only the final test, the target event is different.
• Is the number of tests fixed? The formula is stated for a prespecified \(K\), not an unspecified number of analyses.
• Are the rejection events independent under the null? A common nominal level does not establish independence.
• Does each test reject with probability exactly \(\alpha\)? If it has size below \(\alpha\), using \(\alpha\) gives a bound under independence, not necessarily an exact value.
• Is the null true for all tests in the calculation? Otherwise, not every rejection in the group is a false rejection, and the interpretation changes.

Comparison, Limitations, and a Common Mistake

For a single test, the null rejection probability is \(\alpha\). For \(K\) independent tests, the chance of at least one rejection is \(1-(1-\alpha)^K\). These quantities answer different questions: one is per-test, the other describes the combined set of tests. The calculation here is an exact result under independence; it is not a general formula for every set of repeated tests.

If rejection events are dependent, the probability of no rejections need not equal the product \((1-\alpha)^K\), so the derivation does not establish the same formula. Repeated analyses of accumulating data, for example, require checking whether their outcomes are independent rather than assuming they are. This tutorial’s result should not be transferred to such a setting without justifying the assumption. A single final test is a different alternative: its own null rejection probability is \(\alpha\), but that does not answer the any-rejection question across multiple tests.

A common mistake is to reason, “There are \(K\) tests, each at level \(\alpha\), so the overall probability is \(K\alpha\).” This does not derive the exact probability: it ignores the possibility of multiple rejections in the same set of outcomes. Under independence and exact per-test level, the correct route is to calculate the probability of no rejections and subtract from one. For example, with two tests at level \(0.05\), the exact overall probability is \(0.0975\), not \(0.10\).

The formula also has practical limitations. It relies on a specified number of tests, independence, and the null rejection probabilities used in the calculation. If any of those conditions fail, the numerical result may not describe the procedure. The derivation is still useful because it makes clear which assumptions connect the per-test significance level to the probability of at least one rejection.