Multiple Testing Across Genes and Cell Types

Why the Testing Family Matters

A cell-type QTL scan can involve thousands of genes, many genetic variants, and several cell types. Testing each gene–variant association separately creates a collection of hypotheses, not one isolated test. If each test is judged at the same unadjusted significance level, chance findings become increasingly likely as the number of tests grows. Multiple-testing procedures address this problem, but their meaning depends on which tests are treated as one inferential family.

The family is not determined automatically by the data table or software output. It is the set of hypotheses for which the study intends to make a joint set of claims. A scan may define one family across all gene–variant–cell-type combinations, or distinct families for prespecified cell types or other scientifically justified groups. The choice affects the error rate being controlled and must be stated alongside the results.

A false rejection is a rejection of a true null hypothesis. Family-wise error rate (FWER) control limits the probability of making even one such rejection in the family. False discovery rate (FDR) control instead limits the expected proportion of false rejections among all rejections, defining the proportion as zero when there are no rejections. FDR is not the probability that a particular reported association is false.

Specifying a Family for Cell-Type QTLs

For a scan testing variant–gene associations separately in each cell type, a natural global family contains every association hypothesis actually tested. If \(G\) genes, \(V\) variants per gene on average, and \(C\) cell types are tested, the family has approximately \(GVC\) hypotheses. The exact count should reflect the analysis: for example, whether variants are tested for every gene or only within defined cis windows, whether some tests are absent, and whether several model terms generate separate hypotheses.

The number of tests is not necessarily the number of rows in a report. A row could summarize an omnibus test across variants, or a single association could appear in multiple model outputs. Define the scientific claim first, then count the hypotheses that support that claim. For example, a claim that a variant is associated with a gene in at least one of six cell types has a different inferential scope from six separate claims about association in each named cell type.

Families may be defined separately by cell type when those analyses are genuinely distinct and the intended conclusions are cell-type-specific. However, applying a procedure at level \(\alpha\) separately in each of \(C\) cell types does not, by itself, control FWER at \(\alpha\) across the union of all cell types. If the scientific conclusion pools findings across cell types, global control or a stated allocation of error across the cell-type families is needed. Similarly, conducting a separate gene-level analysis after inspecting results across all cell types does not erase the earlier search.

Family-Wise Error Control

Bonferroni control is simple and valid under arbitrary dependence among tests, provided each true-null p-value is valid: under its null, \(P(p_i \leq t) \leq t\) for every threshold \(t\) between zero and one. It rejects any hypothesis with \(p_i \leq \alpha/m\). Equivalently, its Bonferroni-adjusted p-value is \(\min(1,mp_i)\). The procedure can be conservative, especially when the tests are strongly correlated, as often occurs among nearby variants or related expression traits.

The guarantee follows directly from the union bound. Let \(I_0\) be the set of true null hypotheses, which is unknown in practice. A false rejection occurs if at least one true-null p-value falls below \(\alpha/m\). Therefore,

The first inequality is the union bound, which does not require independent tests. The next inequality uses p-value validity under each true null, and the last uses \(|I_0|\leq m\). This establishes FWER control at level \(\alpha\).

Holm’s step-down procedure also controls FWER under arbitrary dependence with valid p-values and is never less powerful than ordinary Bonferroni. Sort the \(m\) p-values from smallest to largest. Compare the smallest with \(\alpha/m\), the next with \(\alpha/(m-1)\), and continue. Reject in order until the first comparison that fails; do not reject that hypothesis or any larger p-value. This procedure adapts the subsequent thresholds to how many earlier hypotheses have already passed.

False-Discovery Control

In a large exploratory scan, researchers may prefer to identify a broader set of candidates while controlling the expected fraction of false discoveries among the rejected hypotheses. The Benjamini–Hochberg (BH) procedure controls FDR at a chosen level \(q\) when the p-values are independent, and also under certain positive-dependence conditions commonly expressed as positive regression dependence on the subset of true nulls. Dependence among genetic tests should not be casually assumed to satisfy those conditions. For arbitrary dependence, the Benjamini–Yekutieli procedure provides an FDR-control option, typically with a more stringent threshold.

To apply BH, order the \(m\) p-values as \(p_{(1)}\leq \cdots \leq p_{(m)}\). Find the largest rank \(k\) for which \(p_{(k)}\leq kq/m\), and reject the hypotheses at ranks 1 through \(k\). If no rank meets the inequality, reject none. The threshold increases with rank, allowing more discoveries when several p-values are small together. Unlike Bonferroni and Holm, BH controls the expected false-discovery proportion, not the chance of any false rejection.

Choosing and Reporting the Control Strategy

The choice should follow the intended use of the findings. FWER control is often appropriate when even one false association would prompt a consequential confirmatory claim, or when the family is modest in size. FDR control is commonly useful for broad discovery scans where many findings will be followed up, and some false candidates are acceptable if their expected fraction is constrained. Neither target is universally preferable: the family, error rate, and interpretation must match the scientific claim.

For cell-type QTL results, report enough detail to reconstruct the scope of control. State which genes, variants, cell types, and model-based hypotheses were included; whether control was global or applied to separate families; the method and target level; and which adjusted criterion determined rejection. If cell types were analyzed separately, say so explicitly. A list of associations selected from a global scan should not be described as having global error control if the procedure was applied only within each cell type.

The individual tests must also be suitable for multiple-testing procedures. In particular, their p-values must be calibrated under their null hypotheses. A multiplicity correction changes the rejection rule; it does not correct confounding, model misspecification, or a failure of the test’s assumptions. Likewise, testing only a subset chosen after looking at the same results can change the effective family and invalidate a procedure’s stated guarantee unless that selection is accounted for.

Common Mistakes

• Using the number of cell types as the whole family size. If each cell type contains many gene–variant tests, those tests also contribute to the family. Count the hypotheses covered by the intended claim.
• Calling separate within-cell-type control global control. Controlling FWER at 0.05 within each cell type does not imply FWER of 0.05 across all cell types. Use a global procedure or allocate error across families when making a joint claim.
• Interpreting FDR as a probability for each finding. BH controls an expectation across a family of tests; it does not assign each rejected association a probability of being false.
• Assuming correlation makes adjustment unnecessary. Correlated tests may affect power and the suitability of a procedure, but dependence does not remove the need to define the family and control its error rate.