Multiple Testing Across TWAS Genes and Tissues

Why Multiplicity Matters in TWAS

A TWAS analysis can test many genes across many tissues. Even if every null hypothesis is true, a large collection of tests makes it increasingly likely that at least one small p-value appears by chance. The problem is not solved by adjusting only for the number of genes if the reported results also involve multiple tissues, or by treating a gene tested in several tissues as a single test without defining what claim is being made.

The tests are often correlated. Genes can share genetic predictors, prediction models for the same gene can be correlated across tissues, and linkage disequilibrium can contribute to dependence among TWAS statistics. The appropriate adjustment therefore depends both on the scientific family of claims and on what dependence assumptions the procedure requires. Correlation does not make multiplicity disappear, nor does it make every standard correction invalid.

For a gene-by-tissue test, the null hypothesis typically concerns the association between genetically predicted expression for that gene in that tissue and the outcome, under the specified TWAS model. A small p-value is evidence against that test’s null; it does not by itself establish that the gene is causal, that the tissue is uniquely relevant, or that the result is independent of other molecular signals. Those are distinct interpretive questions.

Two Error Rates, Two Scientific Questions

The family-wise error rate (FWER) is the probability of making at least one false rejection in a family of tests. It is a natural target when even one false positive would be costly, for example when a short list of genes will receive expensive follow-up. The false discovery rate (FDR) is the expected proportion of false rejections among all rejections, with the proportion defined as zero when there are no rejections. FDR control is often useful for prioritizing a broader set of candidates for subsequent work.

Here \(V\) is the number of false rejections and \(R\) is the total number of rejections in the family. Controlling FDR at a chosen level does not mean that exactly that fraction of the discoveries will be false in every analysis. It is a repeated-analysis expectation under the procedure’s assumptions.

A useful first step is to state the family before looking at the results. For example, if an analysis evaluates 300 genes in 8 tissues and intends to report any gene-tissue association, the natural family has \(300\times8=2{,}400\) tests. If the stated question instead concerns whether each gene is associated in at least one tissue, a gene-level omnibus test can define a family of 300 tests. These are different hypotheses and require appropriate test statistics; simply choosing whichever family produces more small adjusted p-values is not a principled strategy.

Procedures and Their Dependence Requirements

The Bonferroni procedure controls FWER under arbitrary dependence, provided each individual p-value is valid under its null. With \(m\) tests and target level \(\alpha\), reject a test when its p-value is at most \(\alpha/m\). Equivalently, report the Bonferroni-adjusted p-value \(\min(mp,1)\). Dependence can make this procedure conservative, but does not undermine its FWER guarantee.

The first inequality is the union bound: the probability of at least one event is no greater than the sum of the event probabilities. Here \(\mathcal{H}_0\) is the set of true null hypotheses and \(m_0\) is their number. Since each true-null p-value is super-uniform, \(P(p_i\leq t)\leq t\), the displayed bound holds even when the tests are correlated.

The Benjamini–Hochberg (BH) procedure targets FDR. Sort the p-values as \(p_{(1)}\leq \cdots \leq p_{(m)}\). Find the largest rank \(k\) such that \(p_{(k)}\leq kq/m\), where \(q\) is the target FDR level, and reject the hypotheses ranked \(1\) through \(k\). BH controls FDR under independence and under certain forms of positive dependence, commonly expressed as positive regression dependence on a subset (PRDS) for the true-null test statistics. Correlation alone does not show that PRDS holds: the dependence structure and test construction matter.

When arbitrary dependence must be allowed and a valid joint calibration is unavailable, the Benjamini–Yekutieli (BY) procedure offers FDR control. It uses the more stringent rank thresholds \(kq/(mH_m)\), where \(H_m=\sum_{j=1}^{m}1/j\). This guarantee can be conservative, particularly for large families. A calibrated permutation or resampling procedure can sometimes use the dependence structure more efficiently, but its validity depends on the resampling scheme preserving the relevant null distribution and dependence.

Correlated Prediction Models and Tissue-Level Claims

For the same gene, predicted expression models in two tissues may use overlapping variants and correlated weights. Their predicted expression values, and hence their TWAS statistics, can be strongly correlated. This has two implications. First, a correction procedure that requires independence may not apply automatically. Second, two significant or differently sized marginal associations do not by themselves establish that the underlying tissue associations differ.

The distinction is between testing association within each tissue and testing a contrast between tissues. Let \(\hat{\beta}_1\) and \(\hat{\beta}_2\) be the estimated TWAS effects for the same gene in two tissues, expressed in the same outcome units per standard deviation of genetically predicted expression. A direct difference test uses \(\hat{\beta}_1-\hat{\beta}_2\), with variance \(\operatorname{Var}(\hat{\beta}_1)+\operatorname{Var}(\hat{\beta}_2)-2\operatorname{Cov}(\hat{\beta}_1,\hat{\beta}_2)\). The covariance matters: ignoring it can misstate the uncertainty of the difference.

The covariance in that calculation must be estimated or otherwise justified using the joint analysis; it cannot be inferred from the two marginal p-values alone. Likewise, significance in one tissue and non-significance in another is not itself a test of their difference. A claim of tissue specificity needs an appropriate comparison or interaction test, not a comparison of which marginal p-value crossed a cutoff.

Practical Choices and Common Pitfalls

When TWAS results are reported across genes and tissues, make the family explicit: state whether the adjustment covers all gene-tissue tests, a pre-specified subset, or gene-level omnibus tests. Report the number of tests, the adjustment method, the dependence condition relied upon, and both raw and adjusted p-values where useful. If many analysis choices or tissue subsets were examined before selecting what to report, a narrower family may not reflect the full search that generated the reported claim.

Correlation-aware alternatives require care. Effective-number-of-tests estimates can summarize redundancy, but an estimated effective count does not automatically guarantee FWER or FDR control. Permutation can preserve correlations arising from LD and shared outcomes only if the permutation scheme is valid for the null being tested and respects the data structure. Reported control is only as credible as those assumptions.

• Mistake: “The tests are correlated, so I can ignore multiplicity.” Correlation does not remove the chance of false findings across a family. Use a method with a valid guarantee for the dependence structure or justify a calibrated alternative.
• Mistake: “BH always controls FDR, whatever the correlations.” Its standard guarantee requires independence or an appropriate positive-dependence condition. For arbitrary dependence, BY is one general option, though often conservative.
• Mistake: “A significant result in one tissue and a non-significant result in another proves tissue specificity.” The two marginal tests do not test whether their effects differ. Use a direct contrast that accounts for covariance.
• Mistake: “An adjusted p-value is the probability that this gene is a false positive.” It is a test result adjusted for a specified family, not the probability that the hypothesis is true or that the finding is false.