Multiple Testing Across Genes After Clustering

Why the Testing Family Must Be Defined

After clustering, a common goal is to identify genes that differ between a chosen cluster and the other cells. If many genes are tested, even a collection of valid null p-values will often contain small values by chance. Multiplicity procedures address that chance when the family of claims and the p-value assumptions are specified. They do not, by themselves, make a p-value valid when the same expression data helped determine the cluster labels being tested.

The distinction is important: the testing family determines which multiplicity problem is being controlled, while p-value calibration determines whether the control guarantee applies at all. A family can be carefully defined and adjusted while its p-values remain misleading because of data-dependent clustering.

The definition has to match the claims. If the intended report makes claims for every gene in a fixed one-versus-rest contrast, those gene hypotheses form the family. If the analysis instead makes claims for several cluster contrasts, those are additional claims and cannot be silently ignored when interpreting a guarantee. The family should also reflect any gene inclusion rule: a stated set of genes is not the same family as a larger set searched and then reduced after inspecting results.

A selected partition is the partition produced by the clustering procedure applied to the observed data. Naming a gene-level family after that selection does not turn the partition into a prespecified one. In particular, if the expression values used to test a gene also influenced which cells were grouped together, the ordinary fixed-label reference distribution for its test statistic may not describe the statistic after clustering.

What Multiplicity Adjustments Guarantee

A p-value \(p_g\) is valid for a null hypothesis when, under that null, it is super-uniform: for every \(t\) between 0 and 1, \(P(p_g\leq t)\leq t\). This condition allows discrete or conservative p-values; it rules out an excess of small p-values under the null. To interpret a post-clustering p-value conditionally on the observed selection, the relevant super-uniformity must hold under that conditioning. Validity for a prespecified-label analysis does not automatically imply validity after labels were selected from the same data.

The family-wise error rate (FWER) is the probability of making at least one false rejection in the family. With valid marginal p-values, the Bonferroni procedure rejects \(H_{0g}\) when \(p_g\leq\alpha/m\), where \(\alpha\) is the chosen error-rate level. Equivalently, it reports \(p_{g,\mathrm{adj}}=\min(1,mp_g)\) and rejects when that adjusted value is at most \(\alpha\). No independence between genes is required for this FWER guarantee.

Here \(\mathcal{H}_0\) is the subset of true null hypotheses. The first inequality is the union bound, the second uses validity of each null p-value, and the last uses \(|\mathcal{H}_0|\leq m\). This short justification identifies the key limitation: if clustering has made a null p-value anti-conservative, the second inequality need not hold, so the proof no longer establishes the advertised error rate.

Holm’s step-down procedure also controls FWER under valid marginal p-values and arbitrary dependence. Sort the p-values \(p_{(1)}\leq\cdots\leq p_{(m)}\). Starting with the smallest, compare \(p_{(i)}\) with \(\alpha/(m-i+1)\); reject in order until the first comparison that fails, then stop. Holm is at least as powerful as single-step Bonferroni while retaining the same FWER guarantee under those validity conditions.

A different goal is false discovery rate (FDR), the expected proportion of rejected hypotheses that are true nulls, with the proportion defined as zero if there are no rejections. The Benjamini–Hochberg procedure sorts the p-values and finds the largest \(i\) such that \(p_{(i)}\leq i\alpha/m\), then rejects hypotheses with ranks 1 through \(i\). Its standard FDR guarantee requires valid null p-values and independence or certain forms of positive dependence. It is not a general cure for selection-induced p-value distortion, and arbitrary gene dependence also requires care when invoking its guarantee.

Why Adjustment Cannot Repair Selection Distortion

The Bonferroni and Holm proofs start with valid null p-values. Multiplication, sorting, or thresholding cannot supply a missing validity condition. If a clustering algorithm uses expression patterns to form groups and the subsequent marker test reuses those patterns, a null gene can appear unusually different partly because the grouping was chosen to emphasize differences. The ordinary test may then produce smaller p-values under the null than its reference distribution predicts.

The next example is a deliberately simple probability illustration, not a model of every clustering algorithm. It isolates the logical failure. Suppose a family has \(m=1{,}000\) null hypotheses, but after selection each reported p-value has the distribution \(p_g=U/2{,}000\), where \(U\) is uniform on \([0,1]\), and all family p-values equal this same value. These are not valid p-values: for example, \(P(p_g\leq0.01)=1\), which exceeds 0.01.

Worked Example: Holm’s Procedure on Valid Gene-Level P-Values

For contrast, suppose four hypotheses form a prespecified family and their p-values are valid for the analysis. Let \(\alpha=0.05\), and suppose the ordered p-values are \(0.0004,\ 0.003,\ 0.022,\ 0.08\). Holm’s successive thresholds are

The first comparison is \(0.0004<0.0125\), so reject the first null. The second is \(0.003<0.01667\), so reject the second. The third is \(0.022<0.025\), so reject the third. The fourth is \(0.08>0.05\), so stop and do not reject the fourth. Holm rejects three hypotheses. This conclusion has its FWER interpretation only because the example stipulates valid p-values; if these were post-clustering p-values with unaccounted selection, the same arithmetic would not restore that interpretation.

Diagnostics, Alternatives, and Limits

No single diagnostic proves that post-clustering p-values are valid. Warning signs include marker tests that reuse genes strongly involved in forming the clusters, unusually extreme null results in simulations that rerun the full clustering-and-testing pipeline, or p-value distributions from appropriate negative-control genes that show an excess of small values. A uniform-looking histogram alone is not proof: true effects, dependence, filtering, and the construction of the controls all affect what such a plot should look like.

When the goal is to make ordinary gene-level claims, the main options differ in what they do about selection. The earlier material on sample splitting and independent-dataset validation motivates approaches that separate discovery from testing; selective inference and pipeline-rerunning permutation tests instead require methods that account for the selection process under their own assumptions. Multiplicity adjustment still matters within the resulting valid family, but it does not replace those selection-aware steps.

The practical limitation is that a “gene-level adjusted p-value” is not fully interpretable without the family, target contrast, selection process, and validity assumptions. State whether the family includes all tested genes or only a subset, how that subset was chosen, and whether the gene-level tests account for how the partition was obtained. A smaller adjusted value is not automatically stronger calibrated evidence if the raw p-value’s reference distribution ignored selection.

Common Mistake