Multiple Testing Across Variants And Molecular Traits

Why the Testing Family Matters

A molecular QTL analysis can test thousands of variants against many molecular traits. A small p-value is evidence against one test's null hypothesis, but it does not by itself say how many of the reported findings are false. That depends on the family of hypotheses being considered and on the error rate the analysis is designed to control. For example, adjusting tests separately for each gene addresses a different error question from adjusting all variant–trait tests across the study.

The key challenge is not simply that the tests are numerous. Tests may be dependent because variants are in linkage disequilibrium, molecular traits are correlated, or tests reuse the same participants and measurements. A multiple-testing procedure must therefore be chosen and described in relation to both the family of claims and the dependence conditions it requires.

The FDR is not the probability that any false discovery occurs; that is the family-wise error rate, \(P(V\geq 1)\). When every hypothesis in a family is null, FDR equals the probability of at least one rejection, because any nonzero number of rejections then consists entirely of false discoveries. When some hypotheses are non-null, FDR instead averages the fraction of rejected hypotheses that are false, with a value of zero when there are no rejections.

Benjamini–Hochberg Across a Defined Family

The Benjamini–Hochberg (BH) procedure controls FDR by comparing ordered p-values with ordered thresholds. Let \(p_1,\ldots,p_m\) be p-values for the specified family, and order them as \(p_{(1)}\leq \cdots \leq p_{(m)}\). At target level \(\alpha\), find the largest index \(k\) for which \(p_{(k)}\leq k\alpha/m\). If such an index exists, reject the hypotheses corresponding to \(p_{(1)},\ldots,p_{(k)}\). If none exists, reject none. Each p-value must be valid under its null: in particular, a null p-value is super-uniform, meaning \(P(p_i\leq u)\leq u\) for every \(0\leq u\leq1\).

The BH guarantee depends on the joint behavior of the p-values, not only on each p-value's marginal validity. A standard sufficient condition is that all p-values are mutually independent and the true-null p-values are valid. Under that condition, BH controls FDR at no more than \((m_0/m)\alpha\), where \(m_0\) is the number of true null hypotheses. More generally, a standard sufficient dependence condition is positive regression dependence on the subset of true nulls (PRDS), together with valid true-null p-values. Merely saying that the true-null p-values are mutually independent is not enough: a true-null p-value may still be dependent on the remaining p-values, including non-null p-values, in a way that invalidates the usual bound.

A short argument explains why the independence condition must cover the relationship between null and non-null p-values. For a true-null test \(i\), condition on all the other p-values. Under mutual independence, its p-value remains super-uniform after conditioning. The BH step-up rule implies that the expected contribution of this test to the false discovery proportion is at most \(\alpha/m\). Summing that bound over the \(m_0\) true nulls gives the theorem.

The conditional step is where independence matters: it lets the null p-value retain its super-uniform distribution given the others, while the rejection threshold depends on the whole set of p-values. Without that condition or another suitable dependence condition, the argument does not establish the stated FDR bound. For valid p-values under arbitrary dependence, the Benjamini–Yekutieli procedure uses the more conservative level \(\alpha/H_m\) in the BH thresholds, where \(H_m=\sum_{j=1}^{m}1/j\). This provides FDR control without requiring independence, though often at a substantial cost in power.

Gene-Level and Study-Wide Error Rates

A testing family should match the scope of the scientific claim. In cis-QTL analyses, one possible family consists of all variant–trait tests for one gene or molecular feature. Another consists of all such tests across every gene and feature in the study. Adjusting separately within each gene can control FDR within each gene's family under the procedure's assumptions, but does not generally control FDR or family-wise error across all genes.

A second route is to make one gene-level hypothesis per gene, such as “this gene has at least one cis-QTL association.” The p-value for that hypothesis must be calibrated for the search across variants for that gene. A raw minimum p-value is not generally a valid gene-level p-value: even if every individual p-value is valid, searching many variants makes the minimum small more often under the gene-level null. Permutation calibration or another valid method can account for that within-gene search. Once calibrated gene-level p-values are available, a multiple-testing procedure can be applied across genes. A gene-level rejection supports a claim about the gene, but does not by itself identify which variant is responsible.

Adjusted p-Values and q-Values

A BH-adjusted p-value summarizes the smallest FDR level at which a hypothesis would be rejected by the BH procedure, subject to the procedure's assumptions. For ordered p-values, the adjusted values are calculated from the upper ranks and made nondecreasing down the ordered list. They are often called BH q-values in software, but that name can be confusing: an adjusted p-value is not the probability that a particular reported association is false.

A Storey-style q-value is based on an estimate of the FDR at a p-value threshold. For a threshold \(t\), let \(R(t)\) be the number of observed p-values at or below \(t\). If the null p-values are uniform, the expected number of null p-values at or below \(t\), for a fixed \(t\), is \(m_0t\). Storey's approach estimates the proportion of true nulls, \(\pi_0=m_0/m\), and uses that estimate to estimate FDR at a threshold. A q-value for a test is the minimum estimated FDR over thresholds at least as large as that test's p-value.

One common estimate of \(\pi_0\) uses a tuning value \(\lambda\) and the proportion of p-values larger than \(\lambda\): \(\widehat{\pi}_0(\lambda)=\#\{p_i>\lambda\}/[m(1-\lambda)]\), usually constrained to the interval from zero to one. This estimate can be variable, especially in small families, and the resulting q-values are estimates rather than automatic finite-sample guarantees. Dependence among tests also affects their joint behavior and the uncertainty of data-driven estimates. Valid marginal null p-values give an expected null count no greater than \(m_0t\) at a fixed threshold by linearity of expectation, without independence; however, that fact alone does not make every estimated q-value procedure control FDR under arbitrary dependence. The dependence conditions, estimator, and family must be considered when making a control claim.

Dependence, Reporting, and Common Pitfalls

In QTL scans, positive dependence is plausible: nearby variants can have correlated genotypes, and molecular traits can share biological or technical variation. Dependence does not make a valid marginal p-value invalid, but it can change the performance of a multiple-testing procedure. BH is justified under mutual independence or suitable positive dependence such as PRDS, not merely because the tests are numerous or the null p-values are individually valid. If arbitrary dependence is the intended possibility, a procedure with a corresponding guarantee, such as Benjamini–Yekutieli, is more defensible. Permutation-based calibration can also be useful when it is designed to preserve relevant dependence and match the hypothesis family; it does not remove the need to state what claims the adjustment covers.

A frequent mistake is to say, “BH was applied at 5%, so each selected variant–trait association has at most a 5% probability of being false.” This confuses a repeated-analysis average with a probability attached to one selected result. The correction is to say that, under the stated conditions and for the defined family, the procedure controls the expected proportion of false rejections among all rejections at no more than the target level. Another common mistake is to adjust within each gene and describe the result as study-wide control. The correction is to report the within-gene family explicitly, or to define a study-wide family and adjust across it.