Familywise Error and Multiple Comparisons

Why Several Comparisons Change the Error Risk

A one-way ANOVA omnibus test can provide evidence that not all population means are equal, but it does not identify which means differ. A natural next step is to compare pairs of groups. Each comparison can be tested at a chosen significance level, but carrying out many tests creates more opportunities for a false positive. The chance of at least one such error across the collection is the familywise error rate.

The key is to keep track of what counts as an error. Rejecting a false null hypothesis is not a Type I error. If some pairwise differences really exist, only the tests of the true equal-means hypotheses can produce Type I errors. The number of tests in the family and the number of true null hypotheses in that family are therefore not always the same.

Definition: A family is a specified collection of related hypotheses considered together. The familywise error rate (FWER) is the probability, for the parameter configuration under consideration, of rejecting at least one null hypothesis in the family that is true. A test at significance level \(\alpha_i\) has probability at most \(\alpha_i\) of rejecting its null when that null is true.

For example, suppose a study compares the population means for \(k\) groups, and the planned family consists of every pairwise equality \(H_0:\mu_i=\mu_j\). The number of pairwise tests is \(m=k(k-1)/2\). This is the total number of questions in the family; it is not necessarily the number of true null hypotheses. Let \(I_0\) be the set of indices of true null hypotheses, and let \(m_0=|I_0|\) be their number.

The Probability Calculation Under Independence

For each true null hypothesis \(H_{0i}\), let \(A_i\) be the event that its test rejects \(H_{0i}\). This event is a Type I error. A familywise error occurs if at least one of these events occurs, so its probability is \(P(\bigcup_{i\in I_0} A_i)\). Tests of false null hypotheses do not appear in this union: rejecting one of those hypotheses is not a false rejection.

If the error events for the true null hypotheses are independent, the probability that none of them occurs is the product of their individual non-error probabilities. Taking the complement gives the FWER. If tests use different significance levels, each individual probability of error is at most its level; equality holds when the test has exactly that rejection probability under its null.

\begin{aligned} \mathrm{FWER} &=P\left(\bigcup_{i\in I_0} A_i\right)\\ &=1-P\left(\bigcap_{i\in I_0} A_i^c\right)\\ &=1-\prod_{i\in I_0}P(A_i^c)\\ &\leq 1-\prod_{i\in I_0}(1-\alpha_i). \end{aligned}

The third line uses independence, and the inequality uses \(P(A_i)\leq\alpha_i\). If each true-null test has exact Type I error probability \(\alpha\), the expression is \(1-(1-\alpha)^{m_0}\). Under the complete null, all \(m\) hypotheses are true, so \(m_0=m\), yielding \(1-(1-\alpha)^m\). That familiar formula does not automatically apply to a family containing false nulls.

Important qualification: The product formula requires independence of the error events for the true null hypotheses. Pairwise comparisons commonly share observations or groups, so their test outcomes need not be independent. Without independence, the product formula is not an exact calculation of FWER.

Worked Examples

Worked Example: Three Pairwise Tests Under the Complete Null

Suppose, hypothetically, that three groups have equal population means and that the family consists of all three pairwise tests. Each test is conducted at level \(0.05\). For this illustration, assume the three Type I error events are independent and each has probability exactly \(0.05\). Find the FWER.

All three null hypotheses are true, so \(m_0=3\). For each test, the probability of no Type I error is \(1-0.05=0.95\). By independence, the probability of no false rejection anywhere in the family is

$$ (0.95)^3=0.857375. $$

Therefore,

\mathrm{FWER}=1-(0.95)^3=0.142625\approx 0.143.

The chance of at least one false rejection is about \(14.3\%\), even though each individual test has a \(5\%\) Type I error probability. This is an independence-based illustration, not a claim that the three pairwise tests in a real study are independent.

Worked Example: A Family with Both True and False Nulls

Imagine five groups whose population means have this hypothetical pattern: four means are equal, while the fifth is different. The family includes all ten pairwise comparisons, and each test has level \(0.05\). Suppose, just for the calculation, that the six error events for the true null hypotheses are independent and each has probability exactly \(0.05\).

There are \(\binom{5}{2}=10\) comparisons in total. Among the four equal means, there are \(\binom{4}{2}=6\) true null hypotheses. Each of the four comparisons involving the different fifth mean has a false null hypothesis. Those four tests might or might not reject, but their rejections cannot be Type I errors.

Thus the relevant count for the FWER calculation is \(m_0=6\), not \(m=10\). Under the stated independence assumption,

\mathrm{FWER}=1-(1-0.05)^6 =1-(0.95)^6 =1-0.735092 =0.264908\approx 0.265.

The calculated probability of at least one false rejection among the true nulls is about \(26.5\%\). Using ten in the exponent would incorrectly treat the four false null hypotheses as possible sources of Type I error.

Worked Example: Unequal Levels for the True Nulls

Consider a hypothetical family with four comparisons. Two null hypotheses are true, and the other two are false. The two true-null tests have exact Type I error probabilities \(0.01\) and \(0.05\). Assume their error events are independent. Calculate the FWER.

Only the two true nulls contribute. The probability of no error in the first true-null test is \(1-0.01=0.99\); for the second it is \(1-0.05=0.95\). Independence gives

\begin{aligned} \mathrm{FWER} &=1-(0.99)(0.95)\\ &=1-0.9405\\ &=0.0595. \end{aligned}

So the familywise error probability is \(5.95\%\). The false null hypotheses do not enter the product, even if their tests have different power or significance levels, because rejecting them would not be a Type I error.

Why the Number of Comparisons Matters

When the complete null is true and independent tests each have exact level \(\alpha\), the FWER \(1-(1-\alpha)^m\) increases as the number of tests \(m\) increases. This follows because \(0<1-\alpha<1\), so its \(m\)-th power decreases as \(m\) grows. The risk does not increase by adding a fixed amount for every test; rather, each additional independent test adds another opportunity for error.

In all-pairs comparisons, the number of tests grows quadratically with the number of groups. With six groups, there are \(6(5)/2=15\) pairwise comparisons. If, as a simplified benchmark, all 15 null hypotheses are true and the tests are independent with exact level \(0.05\), then

\mathrm{FWER}=1-(0.95)^{15} =1-0.463291 =0.536709\approx 0.537.

This benchmark gives a \(53.7\%\) chance of at least one false rejection. It illustrates why interpreting each unadjusted \(p\)-value against \(0.05\) as if it were the only test can be misleading when many comparisons are examined. It is not necessarily the actual FWER for real pairwise tests, because those tests are generally related.

Dependence and a General Bound

Pairwise test statistics often share data. For instance, comparisons of group 1 with group 2 and group 1 with group 3 both use observations from group 1. Their error events therefore need not be independent. Depending on the joint pattern of rejections, the actual probability of at least one error may differ from the independent-events calculation; one should not simply insert the total number of tests into \(1-(1-\alpha)^m\) and call the result the exact FWER.

A useful statement that does not require independence is the union bound. Since the event of at least one Type I error is the union of the events \(A_i\), its probability is no greater than the sum of their probabilities:

\mathrm{FWER} =P\left(\bigcup_{i\in I_0}A_i\right) \leq \sum_{i\in I_0}P(A_i) \leq \sum_{i\in I_0}\alpha_i.

The first inequality holds because adding the probabilities of individual events can count outcomes with multiple errors more than once, so the sum cannot be smaller than the probability of their union. This bound may be loose, but it remains valid when tests are dependent. It also reinforces the key distinction: the sum runs over true null hypotheses, not automatically every hypothesis in the family.

Specify the family.
Name the related hypotheses that will be considered together, such as all pairwise mean comparisons among a stated set of groups.

Identify the true nulls for the parameter configuration.
Only rejection of a true null counts as a Type I error. Denote their index set by \(I_0\).

Check the dependence condition.
Use the product formula only if the relevant error events are independent; otherwise, do not present it as an exact FWER calculation.

Interpret the result at the family level.
FWER concerns at least one false rejection somewhere in the specified family, not the error probability of one isolated test.

Why Family Definition Matters

A family is defined by the scientific questions being treated together, not merely by the tests that happen to appear in one software output. If many comparisons are explored and only the smallest \(p\)-values are reported, the reported set can hide the number of opportunities for false positives. A clear analysis identifies the comparisons of interest before interpreting their results and explains what set of claims the error rate is meant to cover.

FWER is also not the probability that any particular rejected result is false, nor is it the probability that the null hypotheses are true. It is a repeated-sampling probability: for a fixed configuration of true and false hypotheses, it describes how often the procedure would make at least one false rejection across the family, under its stated conditions.

Common Mistake

Common mistake: “There are ten pairwise tests at level \(0.05\), so the FWER is \(1-(0.95)^{10}\).” This reasoning assumes all ten nulls are true, all ten error events are independent, and each test has exact error probability \(0.05\). If some nulls are false, their rejections are not Type I errors; if the tests are dependent, the product calculation is not exact. Identify the true nulls and justify independence before using that formula.

Key takeaway: The familywise error rate is the probability of at least one false rejection among a specified family. Under independence, it is \(1-\prod_{i\in I_0}(1-\alpha_i)\) when the true-null tests have exact error probabilities \(\alpha_i\). With a common level \(\alpha\), this becomes \(1-(1-\alpha)^{m_0}\); the exponent is the number of true nulls, and equals the total number of tests only under the complete null.

Check Your Understanding

Answer each question by stating the relevant assumptions where needed.

A family contains four true null hypotheses, and their independent tests each have exact Type I error probability \(0.05\). Calculate the FWER.
A family contains eight tests, but only five null hypotheses are true. Under independence and a common exact test level \(\alpha\), what exponent belongs in the FWER formula, and why?
Explain why rejecting a false null hypothesis does not contribute to the familywise error rate.
Three pairwise tests share observations. Can \(1-(1-\alpha)^3\) automatically be treated as their exact FWER under the complete null? Explain.
State a valid upper bound for FWER when the true-null tests may be dependent and have levels \(\alpha_i\).