Multiplicity Control Across Cell Types

Why Cell-Type Tests Form a Family

A differential-abundance analysis may test whether each of many cell types differs between conditions. Even when every test is individually conducted at level \(\alpha\), examining many tests raises the chance of at least one false positive. The key first step is not choosing an adjustment: it is defining which hypotheses belong to the same testing family.

For example, a study might test one condition contrast for 12 cell types, or test three planned contrasts for each of those types. The latter may involve 36 hypotheses if the scientific claims cover all contrasts. The family should follow the claims the analysis is intended to support, not be chosen after inspecting which p-values are small. Component-wise p-values should come from the donor-level abundance models appropriate to the design; multiplicity adjustment does not make invalid component tests valid.

FWER control is useful when even one false claim in the family would be consequential. FDR control is often useful for screening a larger set of cell types, where some false discoveries may be tolerated in exchange for more power. Neither target is a statement about the probability that a particular rejected hypothesis is false.

In compositional data, cell-type proportions within a sample sum to one. A change in one component therefore constrains the possible changes in the others, and component-wise test statistics or p-values may be dependent. The direction and strength of that dependence depend on the data and fitted model; the sum-to-one constraint does not guarantee that a particular multiple-testing procedure’s dependence assumptions hold.

Family-Wise Error Control

Suppose there are \(m\) null hypotheses, and each corresponding p-value is valid: under its null hypothesis, \(P(p_i \leq t) \leq t\) for every \(0 \leq t \leq 1\). Validity here depends on the component model and experimental design, including appropriate treatment of donors as the independent experimental units. The Bonferroni procedure rejects hypothesis \(i\) when \(p_i \leq \alpha/m\). It controls FWER at no more than \(\alpha\), regardless of dependence among the p-values.

To see why, let \(I_0\) be the set of true nulls. The probability of at least one false rejection is bounded by the sum of the probabilities of rejecting each true null:

This union bound does not require independence. Its tradeoff is that the same stringent threshold is applied to every test, which can cost power.

Holm’s step-down procedure also controls FWER under arbitrary dependence and is never less powerful than ordinary Bonferroni for the same family and \(\alpha\). Sort the p-values as \(p_{(1)} \leq \cdots \leq p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\), then \(p_{(2)}\) with \(\alpha/(m-1)\), continuing in order. Reject sequentially until the first comparison that fails; that hypothesis and all later ones are not rejected. The changing thresholds allow small p-values to pass without requiring every p-value to meet the Bonferroni threshold.

False Discovery Rate Control

The Benjamini–Hochberg (BH) procedure targets FDR rather than FWER. Sort the \(m\) p-values and find the largest rank \(k\) for which \(p_{(k)} \leq kq/m\), where \(q\) is the desired FDR level. Reject the hypotheses at ranks 1 through \(k\). If no rank meets its threshold, reject none. Under independent valid p-values, BH controls FDR at no more than \(q\). It also applies under certain positive dependence conditions, commonly expressed as positive regression dependence on the subset of true nulls (PRDS). Dependence among proportions should not automatically be assumed to satisfy PRDS.

The BH guarantee concerns the expected fraction of false rejections among all rejections, not the probability of any false rejection and not the false-discovery proportion in every particular data set. BH can therefore produce more discoveries than an FWER procedure while allowing a larger chance of at least one false claim.

When arbitrary dependence is a concern and an FDR target is desired, the Benjamini–Yekutieli (BY) procedure uses the more conservative thresholds \(kq/(mH_m)\), where \(H_m=\sum_{j=1}^{m}1/j\). It controls FDR under arbitrary dependence if the component p-values are valid. This conservatism can be substantial. Using BY does not address invalid component tests, such as p-values that fail to account for donor-level dependence.

Dependence, Interpretation, and Practical Choices

Because proportions sum to one, an increase in one cell type’s share must be balanced by decreases in one or more other shares. Consequently, component-wise findings are not independent biological statements. A detected decrease in a relative proportion does not by itself show that the absolute number of cells decreased; it may reflect changes elsewhere in the composition. Multiplicity control governs the error rate for the specified component hypotheses, but it does not turn those relative-abundance results into absolute-count conclusions.

The main choices differ in both the error rate controlled and their assumptions:

A practical analysis should report the family, the component model and p-values, the procedure and target level, and which hypotheses were rejected. Define the family before examining results. State whether it covers only one planned contrast or also additional contrasts, time points, and outcomes. If a broad family is too conservative for the scientific question, the remedy is not to silently narrow it after observing results; instead, distinguish prespecified primary hypotheses from exploratory analyses and describe their error guarantees separately.

Useful checks include confirming that every planned test appears in the family, checking that each input p-value corresponds to the intended null, and verifying the sort order and rank thresholds. Large changes in decisions when the family is expanded or when BY replaces BH are signals to explain the scope and assumptions clearly. Multiplicity correction cannot fix an inadequate number of biological replicates, incorrect model specification, confounding, or underestimated uncertainty. Those issues concern the component tests themselves and require appropriate study design and model assessment.

Common Mistakes

• Wrong: “There are 20 cell types, so the family always has 20 tests.” Correction: Count the hypotheses supporting the claims, including planned contrasts or outcomes when they belong to the same inferential family.
• Wrong: “The proportions sum to one, so BH is automatically valid.” Correction: The constraint implies dependence, but does not establish independence or PRDS. Justify the dependence condition or consider a procedure with the needed guarantee.
• Wrong: “BH at 5% means each reported cell type has at most a 5% chance of being a false discovery.” Correction: It controls the expected false-discovery proportion across repeated applications under its assumptions.
• Wrong: “Multiplicity adjustment makes the analysis robust to model errors.” Correction: Adjustments operate on p-values; they do not repair invalid p-values or convert relative proportions into absolute cell counts.