Why Burden Analyses Need a Testing Family
A burden test can be performed for one gene under one prespecified mask, or repeated across many genes and several masks. Each additional test creates another opportunity to obtain a small p-value by chance. A p-value that would be notable for a single planned test may therefore be much less persuasive when it is the smallest result among dozens or hundreds of analyses.
The solution begins by deciding which claims the analysis will use to answer its scientific question. That set of tests is the testing family. In rare-variant analyses, it may include one burden test for each gene using a prespecified mask, or multiple tests per gene if several masks are examined and any of them could support a reported gene-level finding. The family is defined by the inferential claims being considered, not merely by the number of rows in a final results table.
For example, if a study tests 100 genes and reports any gene with a significant burden association, those 100 tests ordinarily form a family. If the analysis tests five masks in each gene and reports whichever mask gives the smallest p-value, the relevant family may contain 500 tests. By contrast, if one mask was selected in advance and only that mask is used for the confirmatory claim, the other possible masks are not automatically extra tests—provided the observed results did not influence the choice.
Bonferroni Control
Let \(m\) be the number of tests in a family, with valid p-values \(p_1,\ldots,p_m\). Bonferroni rejects a null hypothesis only when its p-value is at most \(\alpha/m\). Equivalently, it reports the adjusted p-value \(\min(1,mp_i)\) for test \(i\), and compares that value with \(\alpha\).
The guarantee does not require the tests to be independent. In a burden analysis, tests can be correlated because genes or masks share variants, or because their burden variables are related. Dependence may affect how conservative Bonferroni is, but it does not invalidate its error-rate guarantee.
The justification is the union bound. Under any configuration of true and false null hypotheses, let \(T\) be the set of true nulls. A valid p-value for a true null satisfies \(P(p_i\leq t)\leq t\). Therefore,
This argument also shows why the correction depends on the family: changing the set of tests changes \(m\), and thus changes the threshold. Bonferroni can be conservative, particularly when many tests are strongly correlated, but it provides a simple and transparent familywise guarantee.
Worked Example: One Prespecified Mask Across 20 Genes
Suppose investigators prespecify one mask and one burden test for each of 20 genes. All tests use valid p-values, and the planned familywise significance level is \(\alpha=0.05\). A hypothetical analysis gives a p-value of \(0.0018\) for Gene A. Determine whether it passes the Bonferroni criterion.
Family and threshold. There are \(m=20\) planned tests, so the threshold is \(0.05/20=0.0025\).
Comparison. Since \(0.0018<0.0025\), the Gene A test passes. Its Bonferroni-adjusted p-value is \(\min(1,20\times0.0018)=0.036\), which is below \(0.05\).
Interpretation. Under the assumptions that the individual p-values are valid and the family was defined as planned, this result is significant with Bonferroni familywise control at 5% across the 20 gene tests. It does not mean there is a 3.6% probability that the Gene A null hypothesis is true.
Holm’s Step-Down Procedure
Holm’s procedure also controls the FWER under arbitrary dependence, while it can be less conservative than applying the Bonferroni threshold to every test. Order the \(m\) p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). Starting with the smallest, compare \(p_{(i)}\) with \(\alpha/(m-i+1)\). Reject in order until the first comparison that fails; do not reject that hypothesis or any of the remaining hypotheses.
The threshold increases as the procedure moves down the ordered list. It is still a familywise procedure, not a set of unrelated tests at level \(\alpha\). Its strong-control justification can be seen by considering the first true null in the ordered list. If there are \(k\) true nulls, at least \(k\) hypotheses remain at that point, so its Holm threshold is at most \(\alpha/k\). A false rejection of any true null requires that this first true null have a p-value at most \(\alpha/k\). By the union bound, the probability that any of the \(k\) true-null p-values is that small is at most \(k(\alpha/k)=\alpha\).
Holm-adjusted p-values can also be reported. For the p-value at rank \(i\), they are \(\min\{1,\max_{j\leq i}[(m-j+1)p_{(j)}]\}\). The running maximum ensures the adjusted p-values do not decrease as the ordered raw p-values increase.
Worked Example: Applying Holm Across Genes and Masks
Suppose a hypothetical exploratory analysis tests six genes under three masks each, giving \(m=18\) valid p-values. The smallest four, in order, are \(0.0004\), \(0.0010\), \(0.0015\), and \(0.0040\); the remaining fourteen p-values are all greater than \(0.0040\). Let \(\alpha=0.05\). Apply Holm’s procedure.
At rank 1, the threshold is \(0.05/18=0.002778\). Since \(0.0004<0.002778\), reject the first null.
At rank 2, the threshold is \(0.05/17=0.002941\). Since \(0.0010<0.002941\), reject the second null.
At rank 3, the threshold is \(0.05/16=0.003125\). Since \(0.0015<0.003125\), reject the third null.
At rank 4, the threshold is \(0.05/15=0.003333\). Since \(0.0040>0.003333\), stop. Holm rejects the first three hypotheses and does not reject the fourth or any later hypothesis.
For instance, the third ordered test has Holm-adjusted p-value \(\max(18\times0.0004,\;17\times0.0010,\;16\times0.0015)=\max(0.0072,0.017,0.024)=0.024\). It is significant at familywise level \(0.05\). The fourth has adjusted p-value at least \(\max(0.024,15\times0.0040)=0.060\), so it is not significant. The family-level interpretation applies to the full set of 18 gene-by-mask tests, not just to the three that passed.
Prespecified and Exploratory Mask Analyses
Whether masks count as multiple tests depends on how the analysis was conducted and what claim is being made. A single mask, threshold, and weighting scheme chosen before examining association results can define a confirmatory test for each gene. If investigators instead try several masks and report the one with the smallest p-value for a gene, the reported result has been selected using the data. The family must account for the tested alternatives that could have led to that report.
A useful practical distinction is between the planned primary analysis and exploratory analyses. The primary analysis should identify the gene tests and mask choices that will support confirmatory claims. Other tested masks can still be informative, but if their results are used to identify discoveries, their multiplicity must be addressed. Calling analyses “exploratory” does not remove the selection; it clarifies that the results need appropriate adjustment and cautious interpretation.
The family should match the scope of the claim. If the stated claim is “at least one of these 30 genes is associated under the prespecified mask,” the family contains those 30 tests. If the claim names a particular gene because it was the strongest result after searching several masks for every gene, that search is relevant to the family even if only one p-value appears in the final sentence. A correction cannot be chosen honestly by counting only the results ultimately reported.
Worked Example: One Gene Searched Under Five Masks
In a hypothetical exploratory analysis, researchers test one gene under five masks and then highlight the mask with the smallest raw p-value, \(0.008\). Assume all five tests have valid p-values and the family consists of these five candidate tests. At \(\alpha=0.05\), calculate the Bonferroni result.
The threshold is \(0.05/5=0.010\). Since \(0.008<0.010\), the result passes. Its adjusted p-value is \(\min(1,5\times0.008)=0.040\), also below \(0.05\).
This is familywise-significant evidence for the selected gene-and-mask test within the five-test family, under the stated assumptions. The adjustment does not turn the selected mask into a prespecified choice, and it does not establish that this mask is biologically superior to the other four. If the mask had genuinely been fixed in advance and only one test were part of the claim, the five-mask correction would not be needed for that single planned test. But that is not the analysis described here: all five were examined and the smallest p-value was selected.
What the Adjustment Does—and Does Not—Guarantee
A multiplicity adjustment controls a specified error rate for a specified family; it does not repair problems with the individual tests. The p-values must be valid under their null hypotheses, including appropriate calibration for the burden-test setting and the analysis design. For example, adjustment does not correct a p-value that is invalid because a model assumption or calibration requirement was not met.
Familywise error control is also different from controlling the false discovery rate (FDR). FWER concerns the probability of at least one false rejection in the family. FDR concerns the expected proportion of false rejections among all rejections. They answer different questions, so a procedure intended for one target should not be described as controlling the other.
Finally, a significant adjusted result is evidence against the tested null hypothesis under the stated model and family definition. It does not show that every variant in the mask is causal, establish a mechanism, or guarantee that the association will replicate. The conclusion should name the gene, mask, trait, adjustment method, and family to which the error-rate statement applies.
Common Pitfall
Wrong reasoning: “We reported only the smallest p-value, so there was only one test and no correction is needed.”
Correction: The relevant count is the set of tests that could have produced the reported claim, not just the number of p-values printed. If several masks or genes were examined and the most favorable result was selected, define the family to include those opportunities and adjust accordingly. Conversely, do not inflate the family by counting options that were genuinely prespecified as unavailable for the confirmatory analysis and were not used to select the reported result.
Check Your Understanding
Answer each question by identifying the family and the relevant error-rate interpretation.
- A study tests 40 genes using one prespecified mask per gene at \(\alpha=0.05\). What is the Bonferroni threshold?
- Why can Bonferroni control familywise error when burden tests are correlated?
- In Holm’s procedure, what happens to the current and subsequent hypotheses after the first ordered p-value fails its comparison?
- A researcher tries four masks for each of 10 genes and reports the smallest p-value. What information is needed to define the relevant testing family?
- In one sentence, distinguish familywise error rate from false discovery rate.