Why Organize Marker Tests as a Hierarchy?
A marker analysis can make two distinct claims. A cluster-level claim says that a cluster has at least one marker among a defined set of genes. A gene-level claim identifies a particular gene as a marker for a particular cluster. The second claim is more specific, and finding evidence for a cluster-wide signal does not identify which gene produced it.
Suppose the analysis considers \(C\) clusters and \(G\) genes. Testing each cluster-gene combination creates \(C G\) gene-level hypotheses. Testing the \(C\) cluster-level hypotheses as well creates a hierarchy: a cluster test is an upper-level screen, while gene tests are follow-ups. The central design question is not merely how to calculate p-values. It is which hypotheses form each testing family and what error probability the procedure controls across the resulting claims.
The p-values supplied to this hierarchy must themselves be valid for the data and analysis that produced them. In particular, clustering a set of cells and then testing those same cells for the genes that helped define the clusters can produce invalid ordinary p-values. The earlier material on sample splitting, selective inference, and clustering-aware permutation tests gives approaches for addressing that selection problem. Hierarchical multiplicity adjustment does not repair invalid input p-values.
Define the Two Levels Before Testing
Let \(H_{cg}\) denote the null hypothesis that gene \(g\) is not a marker for cluster \(c\), for the chosen contrast and target population. For example, the contrast might compare cells assigned to cluster \(c\) with cells outside that cluster. The definition of a marker null must match the estimand and test; it should not be left implicit.
The cluster-level null is the intersection of its gene-level nulls: \(H_c\) says that none of the \(G\) genes in the prespecified set is a marker for cluster \(c\). Its alternative says that at least one is. Rejecting \(H_c\) supports an “at least one marker” claim for that cluster, but does not, by itself, identify any particular gene.
The family definition is substantive. If only some genes are in scope, define that gene set before looking at their p-values and use it consistently. If the scope is all \(C G\) cluster-gene combinations, the child family includes combinations in clusters that are not selected at the first stage. Restricting the family after observing which clusters look promising changes the multiplicity problem.
A Conservative Two-Stage Procedure
Choose a total family-wise error target \(\alpha\), and divide it into two nonnegative budgets, \(\alpha_P\) for parent tests and \(\alpha_G\) for child tests, with \(\alpha_P+\alpha_G\leq\alpha\). Apply Holm’s step-down procedure to the parent family at level \(\alpha_P\). Separately apply Holm’s procedure to the entire prespecified child family at level \(\alpha_G\). Report a gene-level marker only when both its parent test and its own child test are rejected.
Define each cluster contrast, the gene set, all parent nulls, and all \(C G\) child nulls before interpreting the p-values.
Order the \(C\) parent p-values from smallest to largest and apply Holm at \(\alpha_P\). A rejection supports an “at least one gene” claim for that cluster.
Order all \(C G\) child p-values and apply Holm at \(\alpha_G\), not just to p-values from clusters selected in Step 2.
Report \(H_{cg}\) as rejected only if Holm rejects it in the child family and Holm rejects its parent \(H_c\).
For a family of \(m\) hypotheses with ordered p-values \(p_{(1)}\leq\cdots\leq p_{(m)}\), Holm’s procedure compares \(p_{(i)}\) with \(\alpha/(m-i+1)\), beginning at \(i=1\). It rejects hypotheses in order until the first comparison that fails; that hypothesis and all remaining ones are not rejected. If every p-value is valid under its null, Holm controls the family-wise error rate (FWER), the probability of at least one false rejection in that family, at no more than \(\alpha\), under arbitrary dependence among the p-values.
Why the Error Guarantee Holds
The construction gives an overall guarantee for the combined claims, rather than treating the two stages as unrelated opportunities to use the full \(\alpha\). Let \(E_P\) be the event that at least one true parent null is rejected, and \(E_G\) the event that at least one true child null is rejected by the global child procedure. Holm’s strong FWER control gives \(P(E_P)\leq\alpha_P\) and \(P(E_G)\leq\alpha_G\). Any false cluster claim belongs to \(E_P\), and any false gene claim made by the gated procedure must also belong to \(E_G\). Therefore, by the union bound,
This argument does not require independence between parent and child p-values. It does require valid p-values for the hypotheses being tested and a child family fixed independently of the observed child p-values. It is also why one cannot simply test the parents at \(\alpha\), then test every selected cluster’s child family at \(\alpha\), and claim that the whole tree has FWER \(\alpha\). The separate error opportunities need an explicit combined control argument.
A parent p-value can come from a valid cluster-level global test. If only valid gene-level p-values are available, a conservative parent p-value is
Under \(H_c\), all \(G\) gene nulls are true. For any \(t\) between 0 and 1, the union bound gives \(P(p_c\leq t)\leq\sum_{g=1}^{G}P(p_{cg}\leq t/G)\leq t\). Thus \(p_c\) is valid under the cluster-level null, without requiring independent gene tests. This parent test can be conservative; its role is to test the global “any marker” question, not to replace the gene-level procedure.
Worked Example: Holm Testing Across Cluster-Level Hypotheses
Suppose three clusters, A, B, and C, are under study. Their valid parent p-values are \(0.004\), \(0.018\), and \(0.20\), respectively. Set \(\alpha=0.05\), with \(\alpha_P=0.025\) and \(\alpha_G=0.025\). Test the parent family of three hypotheses with Holm at \(\alpha_P\).
The ordered p-values are \(p_{(1)}=0.004\), \(p_{(2)}=0.018\), and \(p_{(3)}=0.20\). Holm’s first threshold is \(0.025/3=0.00833\). Since \(0.004\leq0.00833\), reject the first parent null, for cluster A. The second threshold is \(0.025/2=0.0125\). Since \(0.018>0.0125\), stop; neither B nor C is rejected.
The parent-level conclusion is that the analysis supports an “at least one marker” claim for cluster A. It does not yet support a claim about any particular gene. The allocated parent-family FWER is at most 0.025.
Worked Example: Applying the Child-Family Gate
Suppose there are three clusters and four genes, so the prespecified child family contains 12 hypotheses. Continue with the same allocation, \(\alpha_G=0.025\). Assume the ordered child p-values are \(0.001\), \(0.003\), \(0.006\), \(0.012\), followed by eight values all at least \(0.04\). Suppose the smallest p-value, \(0.001\), is for gene 1 in cluster A.
The first Holm threshold is \(0.025/12=0.002083\). Because \(0.001\leq0.002083\), reject the first child null. The second threshold is \(0.025/11=0.002273\). Since \(0.003>0.002273\), stop. No other child null is rejected. Gene 1 in cluster A passes both gates: the parent null for A was rejected in the preceding example, and this child null was rejected in the pooled child family. It may be reported as a marker under the specified contrast.
The p-value \(0.006\), even if it belongs to a gene in A, does not pass the child-family procedure. Nor would a rejected child null in B or C be reportable here, because neither parent was rejected. The child-family FWER is at most 0.025 across all 12 tests.
Worked Example: Constructing Parent P-Values From Gene Tests
Consider two clusters and three genes per cluster. For cluster A, suppose the valid gene p-values are \(0.004\), \(0.21\), and \(0.70\). For cluster B, suppose they are \(0.030\), \(0.40\), and \(0.80\). Use the Bonferroni parent p-value and set \(\alpha_P=0.03\).
For A, the smallest gene p-value is \(0.004\), so \(p_A=\min(1,3\times0.004)=0.012\). For B, \(p_B=\min(1,3\times0.030)=0.090\). With two parent tests, Holm first compares \(0.012\) with \(0.03/2=0.015\), so it rejects \(H_A\). The next comparison is \(0.090\) with \(0.03\); it fails, so \(H_B\) is not rejected.
This provides parent-level evidence for at least one marker in A, but it does not authorize a gene-level conclusion on its own. The gene p-values must still be tested in the defined child family with the separately allocated child budget. Multiplying a minimum p-value by three here calibrates the parent test; it is not a substitute for the child-family adjustment.
Diagnostics, Alternatives, and Limitations
Before interpreting results, check the hierarchy itself. Confirm that every cluster-level null corresponds to the gene set used to construct its parent test, that the gene-level contrasts are stated, and that the child family includes the planned cluster-gene combinations. Verify that the p-values are valid after the clustering and any selection steps. A multiplicity procedure cannot correct p-values made anti-conservative by double dipping, inappropriate exchangeability assumptions, or an incorrect model.
A common mistake is to select the clusters with the smallest parent p-values, then apply Holm at \(\alpha\) only to the genes in those selected clusters. This treats a data-dependent subset as though it had been fixed in advance; selection can make the child p-values too favorable for that reduced-family calculation. In the procedure taught here, the child family remains all \(C G\) hypotheses, so the gate changes which results may be claimed but not the family used to adjust the child p-values.
This global child-family approach is transparent and controls FWER, but it can be conservative: it spends child-stage error budget across tests in clusters that do not pass the parent stage. A more powerful tree-based or gatekeeping method may use stage-specific weights or structured local levels. Such a method needs its own formal error-control proof; simply reusing \(\alpha\) at each node is not enough. Likewise, false discovery rate procedures target the expected proportion of false discoveries among rejections, not the probability of any false claim. They answer a different error-control question and should not be described as providing the FWER guarantee above.
Finally, a non-rejection is not evidence that a cluster has no markers or that a gene has no effect. It means only that the specified procedure did not reject that null at its allocated level. The result depends on the target population, contrast, chosen genes, clustering strategy, and validity of the underlying tests. Report those choices alongside the family definitions and the error budget.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- What does rejection of a cluster-level global null establish, and what does it not establish?
- If \(\alpha=0.05\), choose one valid equal split of the error budget and state the family tested at each stage.
- Why is adjusting child p-values only within clusters selected by their observed parent p-values not the procedure defined here?
- Suppose the parent-family FWER is controlled at \(0.02\) and the global child-family FWER at \(0.03\). What upper bound follows for the probability of any false claim?
- Can a gene-level null be reported as rejected if its child test rejects but its parent cluster test does not? Explain.
Solutions. (1) It supports the claim that at least one gene in the defined set is a marker for that cluster; it does not identify a particular gene. (2) For example, use \(\alpha_P=0.025\) and \(\alpha_G=0.025\), testing all cluster nulls and all prespecified cluster-gene nulls, respectively. (3) The selected child subset depends on observed results; treating it as fixed can invalidate the adjustment. The defined procedure adjusts across all planned child hypotheses. (4) The union bound gives \(0.02+0.03=0.05\). (5) No. The reporting rule requires rejection of both the child null and its parent, so the gene-level claim is not eligible.