Why Several Model Comparisons Change the Question
A likelihood comparison can be informative when its hypotheses, data, and calibration are specified in advance. The interpretation changes when an analyst examines several comparisons and reports whichever appears most convincing. Even if every individual test has a small false-rejection probability under its null hypothesis, testing repeatedly creates more opportunities for at least one false rejection.
This is a risk of overinterpreting likelihood evidence, not a reason to avoid comparing models. The key is to define the set of claims being considered, or the comparison family, and to control the error rate appropriate to that family. Multiplicity adjustment is a layer on top of the likelihood-ratio or other valid comparison method; it does not repair an invalid likelihood or an unsuitable test calibration.
A planned single comparison has one prespecified null hypothesis and one prespecified test. If its p-value is valid and the test is conducted at level \(\alpha\), its false-rejection probability is at most \(\alpha\). No multiplicity adjustment is required merely because other models exist; the concern arises when multiple results are used to support a conclusion or when the reported comparison is chosen after examining the results.
By contrast, a data-driven search might compare one baseline model with several alternatives, compare every pair in a candidate set, or continue testing until a small p-value appears. The individual p-values do not automatically account for the search. A p-value that is valid for one fixed null and test does not, by itself, provide the same error guarantee after selecting the most favorable result from many tests.
Bonferroni and Holm Procedures
Suppose a family contains \(m\) hypotheses and each test has a valid p-value under its null. The Bonferroni procedure tests each hypothesis at level \(\alpha/m\). Equivalently, it rejects a hypothesis when its Bonferroni-adjusted p-value, \(\min(1,mp)\), is at most \(\alpha\). This procedure controls FWER even when the test statistics are dependent.
Here \(I_0\) is the set of true null hypotheses and \(m_0\) is its size. The first inequality is the union bound. Since \(m_0\leq m\), the final inequality follows. Bonferroni is simple and robust, though it can be conservative.
Holm’s step-down procedure often gives more power while retaining FWER control under arbitrary dependence among valid p-values. Sort the p-values as \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\). If it is no greater, reject that null and compare \(p_{(2)}\) with \(\alpha/(m-1)\); continue in this way. At the first failure, stop and do not reject that hypothesis or any hypothesis with a larger ordered p-value.
List the comparisons that will support the conclusion. Do not define the family after seeing which results look favorable.
Each p-value must be appropriate for its comparison, including the likelihood structure and any required calibration.
Use Bonferroni or Holm for the intended FWER guarantee, and report the method and family size.
Say which comparisons pass the familywise procedure. Do not treat a non-rejection as proof that two models are equivalent.
Why Holm Controls the Familywise Error Rate
Holm’s procedure is a step-down refinement of Bonferroni. To see why it controls FWER, suppose \(m_0\) null hypotheses in the family are true. If any true null is rejected, consider the true null having the smallest p-value among the true nulls. Let its position in the ordered list of all \(m\) p-values be \(j\). There can be at most \(m-m_0\) false nulls before it, so \(j\leq m-m_0+1\). Therefore \(m-j+1\geq m_0\).
For Holm to reject that true null, its p-value must pass the step-\(j\) threshold. Consequently it must be no greater than \(\alpha/(m-j+1)\), which is no greater than \(\alpha/m_0\). Thus, if Holm rejects any true null, at least one of the \(m_0\) true-null p-values is at most \(\alpha/m_0\). By the union bound and validity of each true-null p-value, the probability of this event is at most \(m_0(\alpha/m_0)=\alpha\). This argument does not require the p-values to be independent.
Holm’s first threshold is the same as Bonferroni’s, but later thresholds become less strict if earlier ordered p-values pass. It is therefore never less powerful than the single-step Bonferroni procedure when applied to the same family and p-values.
Worked Example: One Planned Comparison Versus Four Explored Alternatives
An analyst has a prespecified likelihood comparison of a baseline severity model against one alternative. The valid p-value is 0.018, and the chosen test level is \(\alpha=0.05\). Separately, suppose the analyst had instead explored four alternatives against the baseline, obtaining p-values 0.012, 0.031, 0.048, and 0.20. Determine the appropriate conclusions for the two designs using Bonferroni for the four-comparison family.
Planned single comparison. There is one prespecified test, so compare \(0.018\) with \(0.05\). Since \(0.018<0.05\), reject the null for this test at the stated level. This conclusion is conditional on the comparison having genuinely been planned and the p-value being valid.
Four explored comparisons. The family size is \(m=4\). The Bonferroni threshold is $$ \frac{\alpha}{m}=\frac{0.05}{4}=0.0125. $$ Compare each p-value with 0.0125: \(0.012\leq0.0125\), while \(0.031>0.0125\), \(0.048>0.0125\), and \(0.20>0.0125\). Only the first comparison is rejected by this procedure.
The adjusted p-values are \(\min(1,4p)\): \(0.048\), \(0.124\), \(0.192\), and \(0.800\). Only the first is at most 0.05. The two analyses answer different questions: a single test can use the chosen level, while a claim selected from four explored comparisons needs to account for the search. The common mistake is to report the four-family result as significant merely because \(0.012<0.05\); the correction is to compare it with \(0.0125\), or use its adjusted p-value.
Worked Example: Holm Adjustment for All Pairwise Model Comparisons
A hypothetical analyst compares every pair among four candidate models, giving \(m=6\) comparisons. Assume each comparison has a valid p-value under its null, and use Holm’s procedure with \(\alpha=0.05\). The sorted p-values are 0.004, 0.009, 0.018, 0.026, 0.061, and 0.20. Determine which null hypotheses are rejected.
At step 1, compare \(0.004\) with \(0.05/6=0.008333\ldots\). It passes, so reject the first null.
At step 2, compare \(0.009\) with \(0.05/5=0.010\). It passes, so reject the second null.
At step 3, compare \(0.018\) with \(0.05/4=0.0125\). It fails because \(0.018>0.0125\). Holm stops here: do not reject the third, fourth, fifth, or sixth null.
The familywise procedure therefore rejects the two hypotheses associated with p-values 0.004 and 0.009. A common error is to test each of the six p-values against 0.05 and reject the first four. That is a set of unadjusted decisions, not the specified Holm procedure; the step-down stopping rule is essential.
Independence: Exact Probabilities and Upper Bounds
When all null hypotheses are true and the tests are independent, the FWER can be calculated from the individual false-rejection probabilities. Let \(q_i=P(\text{reject }H_i)\). Independence gives the probability of no false rejection as \(\prod_{i=1}^m(1-q_i)\), so
This equality uses the actual probabilities \(q_i\), not just their upper bounds. If each test is conducted at level \(\alpha\), validity only guarantees \(q_i\leq\alpha\). Under independence, the FWER is then at most \(1-(1-\alpha)^m\); equality holds only when every \(q_i=\alpha\). Without independence, that product formula does not generally apply, though the union bound still gives \(\operatorname{FWER}\leq\sum_i q_i\leq m\alpha\).
Worked Example: Exact FWER When Test Sizes Are Below Their Nominal Levels
Under a hypothetical complete-null scenario, three independent comparison tests have actual false-rejection probabilities \(q_1=0.02\), \(q_2=0.04\), and \(q_3=0.01\). Each is no greater than the nominal level \(\alpha=0.05\). Find the exact FWER and compare it with the upper bound obtained by treating all three tests as having size 0.05.
Because the tests are independent and all nulls are true, the probability of no false rejection is $$ (1-0.02)(1-0.04)(1-0.01) =0.98(0.96)(0.99). $$ First, \(0.98(0.96)=0.9408\). Then \(0.9408(0.99)=0.931392\). Therefore, $$ \operatorname{FWER}=1-0.931392=0.068608. $$
If all three tests had actual rejection probability exactly 0.05, their FWER would be \(1-(0.95)^3\). Since \(0.95^2=0.9025\) and \(0.9025(0.95)=0.857375\), this value is \(0.142625\). It is an upper bound here, not the exact FWER. The common mistake is to infer the equality \(1-(1-\alpha)^m\) solely from independent tests conducted at level \(\alpha\); a level-\(\alpha\) test may have actual size strictly below \(\alpha\). The exact calculation uses the \(q_i\) values provided.
What a Multiplicity Adjustment Does Not Do
A correction addresses the chance of at least one false rejection within a defined family. It does not establish that the best-fitting model is true, quantify predictive performance, or guarantee that a selected model will perform well on new data. Nor does it make an invalid comparison valid. For example, if a likelihood-ratio calibration is unsuitable for a comparison, applying Bonferroni to its p-value does not fix that calibration.
The p-values entering a correction must themselves be valid for the particular comparisons. This matters when the same observations are used repeatedly: the resulting p-values are often dependent, but Bonferroni and Holm do not require independence. It also matters when the analyst has used the data to invent or modify candidate models in ways not represented in the stated family. A correction for a fixed list of tests does not automatically cover an unreported, open-ended search.
For a planned comparison, report that it was prespecified and describe the test and its calibration. For a collection of comparisons, report the family, its size, the adjustment, and which comparisons pass. Keep the conclusion narrow: rejecting one null supports evidence against that null under the comparison’s assumptions; it does not by itself prove that the alternative model is correct or that all other candidates are inferior.
Check Your Understanding
For each problem, identify the relevant family and apply the stated procedure. Assume p-values are valid wherever required.
- An analyst explores five alternatives against one baseline and obtains p-values 0.008, 0.011, 0.040, 0.070, and 0.20. At \(\alpha=0.05\), apply Bonferroni. (Answer: \(m=5\), threshold \(0.05/5=0.01\); reject only the comparison with p-value 0.008.)
- Four comparisons have sorted p-values 0.003, 0.014, 0.020, and 0.041. Apply Holm at \(\alpha=0.05\). (Answer: The thresholds are 0.0125, 0.0167, 0.025, and 0.05; all four comparisons pass their stepwise thresholds, so reject all four nulls.)
- Three independent tests are conducted when all three nulls are true, with actual false-rejection probabilities 0.01, 0.03, and 0.04. Find the exact FWER. (Answer: \(1-(0.99)(0.97)(0.96)=1-0.921888=0.078112\).)
- A researcher prespecifies one comparison at level 0.05, then later examines three additional comparisons and wants to report whichever has the smallest p-value. Is testing that selected p-value against 0.05 justified as a familywise procedure? (Answer: No. The data-driven selection involves a family of four examined comparisons; an adjustment such as Bonferroni or Holm is needed for an FWER claim based on that family.)
- Four independent tests are each conducted at level 0.05 under the complete null, but their actual false-rejection probabilities are only known to be at most 0.05. Give an upper bound for the FWER and state whether it must equal that bound. (Answer: \(\operatorname{FWER}\leq1-(0.95)^4=0.18549375\); it need not equal the bound because the actual test sizes may be below 0.05.)