Multiple Testing and False Positives

Why the Number of Tests Matters

In “Type I Error in a Test About Means” and “Alpha as the Type I Error Rate in t Tests,” you learned that a Type I error occurs when a test rejects a true null hypothesis. At a significance level of \(\alpha=0.05\), a test has a 5% long-run chance of a Type I error when its null hypothesis is true. That statement describes one test.

Researchers may test several population means in one study: for example, whether a program affects several different outcomes, or whether mean results differ across many groups. If every null hypothesis is actually true, each test still has its own 0.05 Type I error rate. But the chance that at least one test in the collection produces a false positive can be much larger than 0.05.

“Family” here means the set of tests considered together. The family-wise error rate is not the same as the Type I error rate for an individual test. This distinction is the central idea: keeping each test at \(\alpha=0.05\) does not keep the chance of any false positive across many tests at 0.05.

Calculating the Chance of at Least One False Positive

Suppose all \(m\) null hypotheses are true, every test uses \(\alpha=0.05\), and the tests are independent. The probability that a single test does not produce a false positive is \(1-0.05=0.95\). Independence lets us multiply these probabilities to find the chance that none of the \(m\) tests produces a false positive. Subtract that result from 1 to get the chance of at least one.

At \(\alpha=0.05\), this becomes \(1-(0.95)^m\). As the number of independent tests grows, the chance of at least one false positive grows too. Each test still has a 0.05 Type I error rate; it is the chance across the set that has increased.

The tests might not be independent if they use the same participants or measure outcomes that are closely related. For example, measurements of several similar skills from the same students may tend to move together. In that case, the probability of at least one false positive depends on how the test results are related, not just on \(m\) and \(\alpha\).

There is a useful calculation that does not require independence: if all \(m\) null hypotheses are true and each test has Type I error probability 0.05, the expected number of false positives is \(0.05m\). To see why, count each false positive as 1 and each non-false positive as 0. Each test contributes an expected \(0.05\) to the count, so the expected total is the sum of those contributions. This does not tell us the probability of at least one false positive, and it does not mean we will observe exactly that expected number.

Worked Example: Twelve Tests of Mean Outcomes

Not Every Test in a Study Is a True Null

The calculation needs careful interpretation when some null hypotheses are false. A Type I error can occur only for a true null hypothesis. If a null hypothesis is false, rejecting it is not a false positive; failing to reject it could instead be a Type II error. For the expected number of false positives, count the true null hypotheses, rather than automatically counting every test.

For the probability of at least one false positive, the independence formula applies to the set of true null hypotheses if those tests are independent. It does not require the false null hypotheses to be included in that calculation. In practice, researchers may not know which null hypotheses are true, so these calculations are often illustrations of what can happen under stated assumptions, rather than a way to label a particular result as false.

Worked Example: Twenty Independent Tests

What a Significant Result Does—and Does Not—Show

If one test in a large set has a p-value below 0.05, that result is statistically significant at the 0.05 level for that test. It does not follow that the result is a false positive. The calculations above describe probabilities over repeated testing under assumptions, especially that a particular null hypothesis is true. They cannot determine whether an individual null hypothesis is true.

At the same time, a significant result selected from many tests deserves context. If a study tests many outcomes and highlights only the significant ones, readers may not see how many tests were conducted. An isolated small p-value may be less persuasive as a stand-alone finding when it emerged from a large set of tests than when it came from a single, planned test. Researchers can make the testing process easier to evaluate by explaining which outcomes were planned in advance and reporting the full set of tests, not only the significant results.

This is one reason to distinguish exploratory analysis from a test of a planned claim. Exploring many outcomes can help identify questions for future study, but an interesting result from that exploration can be checked in a new study. A new study does not make the original result automatically true or false; it provides additional evidence. Always describe the result in the context of the design and the population parameter, as emphasized in “Tying the Conclusion to the Original Claim.”

Common Mistakes and AP Exam Tips

• Confusing per-test alpha with family-wise error rate: An \(\alpha\) of 0.05 gives a 0.05 Type I error probability for one test when its null is true. It does not say that the chance of at least one false positive across many tests is 0.05.
• Calling the expected number a probability: For 12 true null hypotheses, \(12(0.05)=0.60\) is an expected count, not a 60% chance of at least one false positive. Under independence, that chance is \(1-0.95^{12}=0.4596\).
• Using the independence formula without checking its assumption: Multiplying the probabilities of no false positives requires independent tests. If outcomes or test results are related, do not claim the formula gives the exact family-wise error rate.
• Counting false nulls as possible Type I errors: A Type I error is rejecting a true null. If only five of eight nulls are true, the expected number of false positives at \(\alpha=0.05\) is \(5(0.05)=0.25\), not \(8(0.05)\).
• Claiming a particular significant result must be false: A higher chance of at least one false positive across many tests does not identify which result, if any, is false. A full-credit answer describes the long-run chance under the stated assumptions and does not label a specific result without evidence.
• Forgetting to state the assumptions: When using \(1-(1-\alpha)^m\), say that the tests are independent and the \(m\) null hypotheses being counted are true. Without those assumptions, the calculation may not apply.

A strong explanation names the number of tests, identifies which null hypotheses are assumed true, states whether independence is assumed, and distinguishes the expected count from the probability of at least one error. For example: “If all 12 null hypotheses are true and the tests are independent, the probability of at least one false positive is \(1-0.95^{12}=0.4596\).”