Why the Testing Family Matters
A TWAS analysis can produce a p-value for every eligible gene in every tissue, and may repeat that analysis for several traits. The resulting tests are usually not independent: tissues can share expression-prediction information, genes can have correlated predicted expression, and traits can be associated with overlapping genetic signals. A multiple-testing plan must therefore begin by specifying which claims the analysis is intended to support, not by selecting a correction after viewing the smallest p-values.
For a single trait, a tissue-specific claim might concern one gene–tissue pair. A claim that a gene is associated in at least one of several tissues has a different testing target. A claim about any association across a set of traits has a broader family again. The family should include the hypotheses that could have been reported as discoveries under the stated analysis plan.
The validity of a correction depends first on valid marginal p-values. This requires appropriate TWAS test calibration under each null, including the assumptions of the chosen association test and suitable handling of its covariates and dependence structure. Multiple-testing adjustment does not repair an invalid test statistic, systematic confounding, or a p-value computed after unaccounted selection.
Family-Wise Error Control
The family-wise error rate (FWER) is the probability of making at least one false rejection in the family. Bonferroni control rejects a hypothesis when its p-value is at most \(\alpha/m\), where \(m\) is the number of hypotheses in the family and \(\alpha\) is the desired upper bound on the FWER. Equivalently, the Bonferroni-adjusted p-value is \(\min(1,mp)\).
This procedure does not require tests to be independent. Its guarantee does require that the individual p-values are valid and that the family has been correctly identified. The guarantee follows directly from the union bound: the event of at least one false rejection is contained in the union of the events of rejecting each true null.
Here \(I_0\) is the set of true null hypotheses and \(|I_0| \leq m\). Holm’s step-down procedure also controls FWER under arbitrary dependence and is never less powerful than the Bonferroni procedure. Sort the p-values as \(p_{(1)} \leq \cdots \leq p_{(m)}\). Starting with the smallest, compare \(p_{(i)}\) with \(\alpha/(m-i+1)\). Reject in order until the first comparison that fails; do not reject that hypothesis or any later one.
Worked Example: Counting the Family Across Genes, Tissues, and Traits
Suppose a hypothetical analysis plans to test 1,200 eligible genes in four tissues for each of two traits. Every listed gene–tissue combination is testable for both traits, and the scientific claim is that there is an association anywhere in this complete analysis. The planned family therefore contains \(m=1{,}200 \times 4 \times 2=9{,}600\) hypotheses.
For FWER control at \(\alpha=0.05\), the Bonferroni threshold is \(0.05/9{,}600=0.000005208\), or approximately \(5.208 \times 10^{-6}\). A hypothetical p-value of \(4.0 \times 10^{-6}\) passes this threshold; a p-value of \(6.0 \times 10^{-6}\) does not. The first result meets the stated Bonferroni criterion for this family, assuming its marginal TWAS p-value is valid.
If the aim had instead been to make a separate, pre-specified set of claims for each trait, each trait could be treated as its own family of \(1{,}200 \times 4=4{,}800\) tests. That choice controls FWER separately within each trait, not across all discoveries from both traits. It is not equivalent to controlling the probability of any false result across the combined analysis.
False Discovery Rate and the Benjamini–Hochberg Procedure
When many discoveries are expected, controlling FWER can be stringent. The false discovery rate (FDR) is the expected proportion of rejected hypotheses that are false, defined as zero when there are no rejections. Benjamini–Hochberg (BH) controls FDR at target level \(q\) under independence and under certain forms of positive dependence among test statistics. If dependence is arbitrary, the usual BH guarantee need not hold.
To apply BH, sort the \(m\) p-values and find the largest index \(k\) satisfying \(p_{(k)} \leq kq/m\). Reject the hypotheses corresponding to \(p_{(1)},\ldots,p_{(k)}\). If no index satisfies the condition, reject none. BH adjusted p-values can be reported alongside raw p-values; the step-up rule itself determines the rejection set at a specified \(q\).
Worked Example: Applying BH to a Small Hypothetical Family
Suppose eight pre-specified gene–tissue–trait tests have sorted p-values \(0.001, 0.004, 0.009, 0.013, 0.021, 0.032, 0.080,\) and \(0.400\). Choose \(q=0.05\). The BH thresholds \(kq/m\), for \(k=1,\ldots,8\), are \(0.00625, 0.01250, 0.01875, 0.02500, 0.03125, 0.03750, 0.04375,\) and \(0.05000\).
At \(k=6\), \(p_{(6)}=0.032 \leq 0.03750\). At \(k=7\), \(p_{(7)}=0.080 > 0.04375\), and \(p_{(8)}=0.400 > 0.05000\). Thus the largest qualifying index is \(k=6\), and BH rejects the first six hypotheses. This is an FDR procedure, not a guarantee that each rejected association has a 95% probability of being true.
Correlation Changes Interpretation, Not the Family
Correlated tests do not automatically invalidate all multiple-testing procedures. Bonferroni and Holm retain their FWER control under arbitrary dependence, provided the marginal p-values are valid. BH is justified under independence or specified positive-dependence conditions; when those conditions are doubtful, one conservative option is the Benjamini–Yekutieli procedure. It applies BH after replacing \(q\) by \(q/H_m\), where \(H_m=\sum_{j=1}^{m}1/j\), and controls FDR under arbitrary dependence for valid p-values. This can be substantially more conservative, especially for large \(m\).
A crucial distinction is between a correction’s error-rate guarantee and the interpretation of a pattern of results. If two tissues yield significant p-values for the same gene, their results are not automatically two independent pieces of evidence. Shared prediction weights, correlated expression estimates, or shared genetic association structure can make the tests dependent. Adjustment addresses a specified family of statistical rejections; it does not establish that tissues are biologically distinct, that a particular tissue is uniquely implicated, or that an association is causal.
Worked Example: Two Highly Correlated Tissue Tests
Consider two hypothetical tissue-specific tests for the same gene and trait, with p-values \(0.012\) and \(0.012\). For a family consisting of these two tests, Bonferroni gives the adjusted p-value \(\min(1,2 \times 0.012)=0.024\), below \(\alpha=0.05\). Thus both hypotheses meet the Bonferroni rejection criterion for this two-test family.
Now suppose the tests are nearly duplicates because the two tissues have almost identical prediction information. The numerical FWER guarantee remains valid, but the two rejections should not be described as two independent confirmations. Bonferroni does not require independence, and its adjusted p-value does not measure how much independent biological evidence each tissue contributes. The analysis supports rejection for both test hypotheses under the stated family; further interpretation of tissue specificity requires evidence and methods suited to that question.
Choosing and Reporting a Strategy
The choice between FWER and FDR depends on the intended use of discoveries. FWER is suitable when even one false claim among a small, consequential set is especially costly. FDR is often useful in broad discovery screens, where the goal is to limit the expected fraction of false findings among the reported results. Neither target is universally preferable; the family and error rate should reflect the scientific question and be stated in advance.
| Approach | What it controls | Dependence and practical use |
|---|---|---|
| Bonferroni | Probability of at least one false rejection in the family | Valid under arbitrary dependence; simple, often conservative |
| Holm step-down | Probability of at least one false rejection in the family | Valid under arbitrary dependence; at least as powerful as Bonferroni |
| Benjamini–Hochberg | Expected false fraction among rejections | Standard guarantee under independence or certain positive dependence |
| Benjamini–Yekutieli | Expected false fraction among rejections | Valid under arbitrary dependence; typically more conservative than BH |
A practical plan begins by enumerating eligible genes, tissues, and traits, then stating whether the target claim is trait-specific, gene-level, tissue-specific, or global across the analysis. Filtering can reduce the number of tests only when the eligibility rule is defined without using the association evidence in a way that invalidates the remaining p-values. For example, an eligibility filter based on independent expression-prediction quality information is different from keeping only tests with small TWAS p-values and correcting as though the others had never been considered.
If the scientific target is a gene associated in at least one tissue, that is a gene-level claim, not simply the same claim repeated for every tissue. One conservative way to form a gene-level p-value from \(T\) tissue p-values is \(p_g=\min(1,T\min_t p_{gt})\), a Bonferroni bound that remains valid under arbitrary dependence if each tissue p-value is valid. Gene-level p-values can then be adjusted across the genes in the defined family. A gene-level rejection does not, by itself, identify which tissue is associated. Tissue-specific follow-up requires its own stated and valid multiplicity strategy; selecting genes first does not license unadjusted claims about their tissues.
The same principle applies across traits. If discoveries across several traits may be presented as one combined set, include those trait-specific tests in the family or use a procedure explicitly designed to control the desired error rate across the combined claims. Separate corrections can be appropriate for genuinely separate, pre-specified questions, but their guarantees should be described as separate. Report the number and definition of tests, the adjustment method, the target level, and whether the results are gene-, tissue-, or trait-specific.
Diagnostics, Limitations, and Common Pitfalls
Before interpreting adjusted results, check that the reported test count matches the analysis plan, that every p-value corresponds to a valid test, and that no tissue or trait was silently omitted after inspecting results. Check whether BH’s dependence conditions are defensible or whether a more conservative dependence-robust procedure is needed. Also verify that the adjustment corresponds to the claim in the report: a correction performed separately by trait cannot support a claim of FWER control across all traits.
A tempting shortcut is to estimate an “effective number of independent tests” from correlations and substitute that number into Bonferroni. Correlation can make a Bonferroni correction conservative, but an informal effective-test estimate does not automatically provide a valid error-rate guarantee. Use a method with stated assumptions and a demonstrated guarantee, rather than treating a correlation summary as a correction by itself. Permutation-based adjustments can be useful when their null-generating and exchangeability assumptions are appropriate, but those assumptions must also be justified.
Multiplicity adjustment also has limits. It does not fix poor TWAS calibration, address confounding, make correlated findings independent, or distinguish association from causation. Conversely, a non-rejection does not establish absence of association: correction can reduce power, particularly when the family is large or tests are highly correlated. The defensible conclusion is tied to the tested hypothesis, the defined family, the procedure, and its assumptions.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A study tests 500 genes in three tissues for one trait and claims any discovery across the whole analysis. How many tests belong to the family?
- Why can Bonferroni control FWER when tissue tests are correlated?
- At \(q=0.05\), a BH analysis has \(m=100\) tests and its 4th ordered p-value is \(0.0018\). Does this index meet its BH threshold?
- What does an FDR target of 0.05 say, and what does it not say, about a particular rejected test?
- A gene passes a gene-level test combining p-values across tissues. Does that establish which tissue is associated?
Solutions. (1) The family has \(500 \times 3 \times 1=1{,}500\) tests. (2) The union-bound argument sums valid marginal probabilities and does not assume independence. (3) The threshold is \(4(0.05)/100=0.002\); since \(0.0018 \leq 0.002\), the 4th index meets the BH inequality, though the final rejection set is determined by the largest qualifying index. (4) It targets the expected proportion of false rejections among rejected tests under the procedure’s assumptions; it does not give the probability that any particular rejected null is true. (5) No. It supports the gene-level claim of association in at least one tested tissue under the gene-level procedure, but tissue-specific claims require appropriate follow-up testing and multiplicity control.