Why a Family of Tests Needs Its Own Error Rate
A statistical conclusion may involve more than one hypothesis. After comparing several treatment means, for example, an analyst might ask which pairs differ. Each individual test can have a specified Type I error rate, but that rate alone does not describe the chance of making at least one false rejection across all the comparisons. The familywise error rate addresses that broader risk.
The family is not simply “every test in the study.” It is the set of hypotheses whose conclusions are intended to be interpreted together for a defined scientific question. Identifying it requires a decision about scope: which comparisons belong together, and which possible false claims count as a family-level error?
The definition counts only false rejections. Rejecting a false null hypothesis is not a Type I error, and failing to reject a false null hypothesis does not contribute to FWER. If all null hypotheses in the family are true, any rejection is false; if only some are true, only rejections among that true subset count.
What Belongs in the Family?
In a one-way analysis of variance with four groups, the six pairwise mean comparisons form a natural family if the goal is to report which group means differ. By contrast, a study could have several prespecified questions—such as comparing each treatment with a control and separately comparing outcomes across two time periods. Whether those questions form one family or more than one depends on the inferential claims intended and should be decided from the study question, not chosen after seeing which tests are significant.
A family can also be a subset of the tests that were run. If an analyst plans to make claims only about three specified contrasts, those three hypotheses can define the family even if other descriptive analyses appear in the report. But omitting an inconvenient comparison after examining its result does not make the remaining error calculation a sound account of the original set of claims.
The family definition is substantive, not merely a count of tests. Two analysts can legitimately define different families for different goals, but they should state what claims each family covers. An omnibus test of equal means and follow-up comparisons answer different questions: the omnibus test concerns whether all means are equal, while the comparisons identify particular differences. Performing one test does not, by itself, define or control the familywise error rate of other tests.
Worked Example: Defining a Family of Pairwise Comparisons
A hypothetical experiment compares the mean calibration error for three sensor designs, A, B, and C. The planned report will make a separate claim for each pair: A versus B, A versus C, and B versus C. Identify the family and the number of possible false rejections.
Method. List the distinct pairwise null hypotheses specified by the reporting goal. A pairwise null states that the two population means are equal.
The hypotheses are \(H_{0,AB}:\mu_A=\mu_B\), \(H_{0,AC}:\mu_A=\mu_C\), and \(H_{0,BC}:\mu_B=\mu_C\). Thus the family contains three hypotheses. If all three equalities are true, any rejected comparison is a false rejection, and between zero and three false rejections are possible.
The number three follows from the planned claims, not from the number of groups alone as a universal rule. If the study’s sole inferential goal were a single prespecified comparison of A with C, that one hypothesis—not all three possible pairs—would define that narrower family.
Check. There are exactly three unordered pairs among three designs: AB, AC, and BC. No pair is counted twice.
Why Repeated Unadjusted Tests Raise the Risk
Suppose \(m\) hypotheses are all true, and each test has Type I error probability \(\alpha\). If the test outcomes are independent under this global null, the probability that none rejects is \((1-\alpha)^m\). The complement is the probability of at least one rejection:
This is not the same as \(m\alpha\). The sum \(m\alpha\) is the expected number of false rejections under these assumptions, and is also an upper bound on the probability of at least one false rejection. But two or more tests can reject together, so adding their individual probabilities counts some outcomes more than once.
The independence formula follows by taking the complement. “No false rejection” means that every individual true null is not rejected. Under independence, the probabilities of those \(m\) non-rejection events multiply. Subtracting that probability from one gives the probability of at least one rejection:
For dependent tests, the probability of simultaneous non-rejection is not generally a product. This matters in comparisons of group means: pairwise tests often share observations and a pooled error estimate, so their outcomes need not be independent. The union bound remains valid, but it can be loose. Consequently, the number of tests and their individual error rates alone do not usually determine the exact FWER.
Worked Example: Three Independent Tests Under the Global Null
A hypothetical quality study conducts three independent tests of three prespecified process effects. Assume all three null hypotheses are true. Each test has exact Type I error probability \(\alpha=0.05\), and the three test decisions are independent under this global null. Find the probability of at least one false rejection.
Method. Use the complement of the event that all tests avoid rejection. For each test, the probability of no false rejection is \(1-0.05=0.95\).
By independence, the probability that none of the three tests rejects is \(0.95^3=0.95\times0.95\times0.95=0.857375\). Therefore,
The familywise error rate is \(0.1426\), or \(14.26\%\), to four decimal places for the probability. It exceeds the individual \(5\%\) error rate. The sum \(3(0.05)=0.15\) is an upper bound, not the exact answer here.
Check. The result is between \(0.05\), the risk from one test, and \(0.15\), the union-bound upper limit. The calculation depends on the stipulated independence; it is not automatically the exact FWER for three pairwise ANOVA comparisons.
True and False Null Hypotheses in One Family
When some null hypotheses are false, FWER concerns only those that are true. If the true-null tests are independent and each has exact size \(\alpha\), the same complement argument applies to the number of true nulls, not the total number of hypotheses. This yields the FWER at that particular configuration of true and false hypotheses.
A procedure has strong FWER control at level \(\alpha\) if its FWER is at most \(\alpha\) for every configuration of true and false null hypotheses. Weak FWER control refers to control only under the global null, when all null hypotheses in the family are true. These terms describe the range of parameter configurations for which the error guarantee holds; they do not specify a particular adjustment procedure.
Worked Example: Only Some Null Hypotheses Are True
A hypothetical evaluation has five prespecified tests, each with exact Type I error probability \(0.04\) when its null is true. Suppose two null hypotheses are false and three are true. Assume the decisions for the three true-null tests are independent. Find the FWER for this configuration.
Method. Count the true nulls, because rejecting either of the two false nulls is not a Type I error. There are three chances for a false rejection, not five.
For each true null, the probability of not rejecting is \(1-0.04=0.96\). By the stated independence assumption, the probability of no false rejection among the true nulls is \(0.96^3=0.96\times0.96\times0.96=0.884736\). Thus
The FWER for this configuration is \(0.1153\) to four decimal places. The two false null hypotheses do not enter this calculation: rejecting them cannot create a Type I error.
Check. Using all five tests in the independence formula would incorrectly count rejection of a false null as a familywise error. The calculation uses the three true nulls only.
Dependence Can Change the Exact Risk
Repeated testing increases the opportunity for a false rejection, but it does not imply that the independent-test formula applies. Dependence can make rejection events overlap substantially. For example, if two tests have exactly the same rejection event and each has Type I error probability \(0.05\), the probability of at least one rejection is \(0.05\), not \(1-(0.95)^2=0.0975\). The second test adds no new rejection event in this special case.
Worked Example: Two Perfectly Dependent Rejection Events
Suppose two valid tests address true null hypotheses, and under the joint null their rejection events are identical: whenever the first test rejects, the second does too, and vice versa. Each test has Type I error probability \(0.05\). Find the FWER and compare it with the independent-tests calculation.
Method. The event “at least one test rejects” is the union of the two rejection events. Since the events are identical, their union is just that same event.
Therefore, \(\operatorname{FWER}=0.05\). If the tests instead were independent, the FWER would be \(1-(1-0.05)^2=1-0.95^2=1-0.9025=0.0975\).
Check. Both tests still have a \(0.05\) marginal Type I error rate, but dependence changes the probability of their union. The example shows why one cannot calculate exact FWER from the per-test level and test count alone.
Common Mistakes and Practical Interpretation
- Equating per-test error with familywise error. A \(5\%\) Type I error rate for each test does not mean the chance of at least one false rejection across many tests is \(5\%\).
- Applying the independence formula without checking independence. Pairwise mean comparisons from a common ANOVA can be dependent. The product formula requires independence of the relevant test decisions under the null configuration.
- Counting every rejection as an error. FWER counts false rejections among true nulls, not rejections of false nulls or all statistically significant results.
- Defining the family after looking at results. The family should reflect the claims being considered together. Selecting only favorable tests after examining outcomes obscures the scope of the error risk.
- Treating a small omnibus-test p-value as a family definition. An omnibus result and a set of pairwise claims are different inferential questions. The latter still require a clearly stated family and an account of their joint error risk.
For an actuarial or scientific report, the practical first step is to name the hypotheses that support the related conclusions. Then distinguish the per-test significance level from the probability of any false conclusion in that family, and state any assumptions used to calculate the latter. This framing is aligned with the SOA VEE Mathematical Statistics syllabus.
Check Your Understanding
For each problem, identify the relevant family or true nulls before calculating.
- A laboratory plans four independent tests, all null hypotheses are true, and each test has exact Type I error probability \(0.03\). Find the FWER. (Answer: \(1-0.97^4=0.1147\) to four decimal places.)
- A three-group study has planned comparisons A versus B and A versus C only. Both null hypotheses are true; the tests are independent and each has size \(0.05\). What FWER applies to this family? (Answer: \(1-0.95^2=0.0975\).)
- A family contains six hypotheses, of which four are true. The four true-null tests are independent and each has exact size \(0.02\). Find the FWER for this configuration. (Answer: \(1-0.98^4=0.0776\) to four decimal places.)
- Two tests have identical rejection events under their true nulls, and each has Type I error probability \(0.04\). What is their FWER, and what would it be under independence instead? (Answer: \(0.0400\); under independence, \(1-0.96^2=0.0784\).)
- Five hypotheses form a family; two are false and three are true. Assume the three true-null tests are independent and each has size \(0.05\). Find the FWER for this configuration, and explain why five is not the exponent. (Answer: \(1-0.95^3=0.1426\) to four decimal places; only the three true nulls can produce false rejections.)