Comparing FWER Procedures in a Worked Study

Why Compare Procedures on the Same Tests?

In genomic analyses, many hypotheses may be tested on related outcomes, markers, or model contrasts. The resulting test statistics can be correlated, so different familywise error rate (FWER) procedures may give different adjusted p-values even when they analyze the same tests. Comparing them on one fixed set of results helps separate two questions: what each procedure calculates, and what assumptions make its error control reliable.

Consider a hypothetical study testing five prespecified genetic associations. The family is these five hypotheses, and the significance level is \(\alpha=0.05\). Suppose their marginal p-values, ordered from smallest to largest, are \(0.004,\ 0.012,\ 0.028,\ 0.061,\) and \(0.200\). The tests are correlated. These numbers are illustrative, not observations from a real study.

An adjusted p-value for a hypothesis is the smallest significance level at which the procedure would reject that hypothesis, accounting for the family. At a chosen level \(\alpha\), reject when the adjusted p-value is at most \(\alpha\). Marginal p-values alone do not specify how a multiple-testing procedure handles the family.

Bonferroni and Holm: Dependence-Robust Comparisons

Bonferroni rejects a hypothesis when its marginal p-value is at most \(\alpha/m\). Equivalently, its adjusted p-value is \(\min(1,mp_i)\). It controls FWER when each marginal p-value is valid under its null; it does not require independence among tests. This follows from the union bound: the probability of one or more false rejections is no greater than the sum of their individual probabilities.

Holm's step-down procedure orders the p-values \(p_{(1)}\leq\cdots\leq p_{(m)}\), then compares \(p_{(i)}\) with \(\alpha/(m-i+1)\), proceeding from the smallest. It stops at the first failure; that hypothesis and all later ones are not rejected. Its adjusted p-values are the running maxima of \((m-i+1)p_{(i)}\), capped at 1. Like Bonferroni, Holm controls FWER under arbitrary dependence when the marginal p-values are valid. It is never less powerful than Bonferroni in the sense that it rejects at least the hypotheses Bonferroni rejects for the same family and level.

Šidák: A Dependence Assumption Changes the Calibration

For \(m\) independent tests with valid p-values, Šidák's single-step threshold is \(1-(1-\alpha)^{1/m}\). Its adjusted p-value for a marginal p-value \(p_i\) is \(1-(1-p_i)^m\). The derivation uses independence: if each test has probability \(c\) of a false rejection under its null, the probability of no false rejection across \(m\) true nulls is \((1-c)^m\). Setting \(1-(1-c)^m=\alpha\) gives \(c=1-(1-\alpha)^{1/m}\). With a mixture of true and false nulls, the same independence argument bounds the probability of rejecting at least one true null by the corresponding all-true-null probability.

This calibration is not justified merely because the number of tests is known. Correlated test statistics generally produce dependent p-values. Under dependence, the Šidák formula need not control FWER at the stated level. Bonferroni and Holm are available when arbitrary dependence must be allowed and marginal p-values are valid.

Permutation maxT: Calibrating to the Joint Test Statistics

Permutation maxT uses the joint behavior of test statistics rather than assuming independent marginal p-values. Let \(T_j\) be the test statistic for hypothesis \(j\), and suppose large absolute values provide evidence against the null. For each permutation replicate \(b\), calculate the statistics for all tests using the same permitted rearrangement, then record the maximum absolute statistic, \(\max_j|T_j^{*(b)}|\). The single-step adjusted p-value for an observed statistic \(|T_j|\) is the proportion of these permutation maxima at least as large as \(|T_j|\), with an appropriate finite-resampling correction.

The joint resampling must be valid for the experimental design and null hypotheses. For example, labels may be permuted only when the design supports that exchangeability, and any required nuisance structure must be handled by a valid method. Single-step maxT controls FWER under the complete null when the joint null distribution is correctly calibrated. Strong control, covering configurations in which some nulls are false, requires additional justification; subset pivotality is one standard condition used for maxT procedures. It says, in effect, that the joint distribution of the statistics for any subset of true null hypotheses is the same regardless of which other hypotheses are false.

Step-down maxT orders the observed statistics from most to least extreme. At each rank it compares the observed statistic with the permutation maximum over the hypotheses still under consideration. This can yield smaller adjusted p-values than single-step maxT because the step-down comparison does not keep already-removed hypotheses in every later maximum. Strong control still depends on valid resampling and the relevant conditions, including subset pivotality in the usual maxT justification. The earlier material on step-down maxT and subset pivotality gives the procedure's general framework; here the focus is on comparing its results with alternatives.

Reading the Comparison and Checking Assumptions

The results for the same five marginal p-values can be summarized as follows. A rejection is based on an adjusted p-value no greater than \(0.05\).

The table is not a universal ranking of methods. Holm improves on Bonferroni without requiring independence. Šidák may be useful when independence is defensible, but its results here do not supply a valid correlated-test comparison by themselves. MaxT can use the observed dependence structure to calibrate joint extremes, but only when the resampling scheme and strong-control conditions are appropriate. The permutation-based adjusted p-values are also subject to Monte Carlo uncertainty; with only 999 replicates, the attainable adjusted p-values lie on a grid of increments \(1/1000\).

Diagnostics should therefore examine the analysis that supports each row, not just the numerical ordering of adjusted p-values. Check whether the family was specified before interpreting results, whether each marginal p-value is valid, and whether dependence assumptions are plausible. For permutation methods, verify the exchangeability and design restrictions, confirm that the same valid permutation is applied jointly across tests, and use enough replicates to resolve the desired tail probabilities. If permutations violate the design, a precise-looking maxT p-value does not repair the invalid calibration.

Common Pitfalls and Limitations

• Confusing the error being controlled. FWER concerns making at least one false rejection, where a false rejection is rejection of a true null hypothesis. It is not the probability of rejecting a false null.
• Using Šidák because tests are merely numerous or correlated. The standard formula relies on independence. Correlation is not evidence for that condition.
• Assuming permutation automatically gives strong control. A valid complete-null permutation calibration alone does not establish control under every configuration of true and false nulls. State and justify the additional condition used.
• Comparing adjusted p-values without comparing procedures' assumptions. A smaller number is not meaningful evidence of better control if the method's calibration does not apply.
• Ignoring finite permutation resolution. The add-one estimate cannot report arbitrarily small p-values from a limited number of replicates.

No procedure removes the need to define the testing family and justify the underlying tests. FWER methods can be conservative when many hypotheses are tested, while dependence-aware procedures may gain efficiency at the cost of stronger modeling or resampling assumptions. The appropriate comparison is therefore between valid procedures for the actual family and design, not simply the method with the most rejections.