Choosing a strategy starts with the claim
After clustering, a gene can look differentially expressed between groups partly because the same expression data helped create those groups. The practical question is not simply which method is most powerful. It is which method can support the intended claim, given how the groups were formed, what independent samples are available, and what computation can be completed. A strategy that is valid for a fixed set of groups may not be valid for a claim that accounts for selecting those groups.
The main options considered here are data splitting, independent validation, selective inference, and full-pipeline permutation. They do not automatically estimate the same target. For example, a held-out test can assess expression differences for labels assigned by a frozen clustering or label-transfer rule. A selective procedure can instead condition on a specified selection event. A permutation procedure tests a null hypothesis under a specified exchangeability scheme and pipeline. Choose the method after stating the claim and the null, not before.
The independent units matter. Cells from one donor can share biological and technical features, so a split that puts cells from the same donor in both parts does not necessarily create independent validation. When the claim concerns generalization across donors, the split or validation cohort should generally be formed at the donor level. A large cell count cannot compensate for having very few independent biological samples.
What the four strategies require
| Strategy | What is separated or conditioned on? | Main requirements and costs |
|---|---|---|
| Data splitting | Use one part to discover clusters and another to test markers. | Requires enough independent data in both parts and a way to assign test observations using a rule fixed without their marker outcomes. |
| Independent validation | Discover in one cohort and evaluate in a separate cohort. | Requires genuinely independent validation data, compatible measurements, and a defensible way to define or transfer groups. |
| Selective inference | Condition inference on a specified selection event. | Requires a valid model and selection description, plus a method that can handle the actual clustering and selection steps. |
| Full-pipeline permutation | Compare the observed statistic with statistics from null-consistent permuted data after rerunning the pipeline. | Requires exchangeability under the null and enough computation to rerun all relevant adaptive steps. |
In data splitting, the discovery data determine the clustering, candidate markers, and any assignment rule for the test data. Testing then uses the reserved observations. Conditional on the discovery data, the test observations must be independent of the discovery process under the design. If the test expression data are used to change clusters, choose markers, or tune the assignment rule before testing, the separation has been compromised.
Independent validation uses a distinct cohort rather than a reserved subset of the discovery cohort. It is especially useful when the target is reproducibility in new biological samples. Independence does not guarantee comparability: platform, tissue processing, donor composition, and other distribution shifts can change expression or cluster composition. Validation should apply a sufficiently specified group definition or a label-transfer procedure fixed without using the validation marker result.
Selective inference aims to account for the selection event mathematically. Its validity depends on defining the event and using a model and inferential calculation appropriate to the procedure that selected the clusters or genes. It can retain more data for inference than a simple split, but it is not a generic correction that makes any complex pipeline valid. If the selection event cannot be represented or the required distribution cannot be computed reliably, the method may not be usable.
A full-pipeline permutation applies a transformation justified under the null, reruns the relevant adaptive steps, and recalculates the target statistic. Rerunning clustering matters: permuting after clustering while leaving the selected groups fixed does not reproduce the selection process. Exchangeability is not automatic, particularly with donor structure or covariates. The permutation scheme must respect the design and the null being tested. The earlier material on exchangeability and full-pipeline permutation develops these requirements in detail.
A short justification for honest separation
Suppose the data are divided into discovery data \(D\) and test data \(T\). Let \(G(D)\) be the cluster-assignment rule learned using \(D\), and let \(p(T;G(D))\) be a valid test p-value for the stated null when the rule is fixed. If \(T\) is independent of \(D\) under the sampling design, then for each fixed discovery result \(D=d\), the rule \(G(d)\) is fixed. Consequently, if the test is calibrated for that rule,
Averaging this conditional bound over the possible discovery results gives
Thus, separation protects the test from the particular discovery result, provided the test is calibrated and the assignment rule is not adapted using \(T\). This argument does not fix dependence between cells that the split treats as independent, nor does it address multiplicity across many genes; the testing procedure must still account for the family of claims being made.
Worked Example: A donor-level split with a limited sample
Worked Example: A donor-level split with a limited sample
Suppose a hypothetical study has 12 donors, each with 500 measured cells. The investigators want to discover clusters and test whether a candidate gene differs between two groups in new donors. They consider reserving 4 donors for testing and using 8 for discovery.
The total number of cells is \(12\times500=6{,}000\). A donor-level split gives \(8\times500=4{,}000\) discovery cells and \(4\times500=2{,}000\) test cells. These cell totals do not mean the test has 2,000 independent biological replicates: it has only 4 independent donors for a donor-generalization claim.
The discovery step must define a cluster assignment rule that can be applied to the held-out donors. If the clusters are defined only by rerunning clustering jointly on all 12 donors, the held-out labels are no longer independent of the discovery process. A frozen label-transfer rule could provide the required assignment, but it must be specified without using the held-out candidate-gene test result.
Decision. The split is conceptually suitable for honest evaluation in new donors, but four test donors may give imprecise evidence. The investigators should compare this loss of precision with the availability of additional independent donors, a suitable selective procedure, or another design. Treating all 2,000 test cells as independent would exaggerate the information available for the donor-level claim.
Worked Example: Independent validation and a defined estimand
Worked Example: Independent validation and a defined estimand
Consider a hypothetical discovery cohort of 30 donors and an independent validation cohort of 15 donors. A gene and two-group marker contrast are chosen in discovery. The intended validation claim is the difference in mean donor-level expression between the two transferred groups. For illustration, suppose the validation donor-level means are 2.4 and 1.8 normalized expression units, with 8 and 7 donors in the groups.
The estimated contrast is \(2.4-1.8=0.6\) normalized expression units. If the corresponding donor-level sample standard deviations are 0.8 and 0.7, the Welch standard error is \[ SE=\sqrt{\frac{0.8^2}{8}+\frac{0.7^2}{7}} =\sqrt{0.08+0.07} =\sqrt{0.15} \approx 0.3873 \] normalized expression units. The resulting test statistic is \(t=0.6/0.3873\approx1.549\).
This calculation illustrates the scale of uncertainty, not a complete claim of significance: a two-sided p-value and degrees of freedom would be needed for that. More importantly, the calculation is appropriate only if donors are the independent units and the group assignment and contrast were fixed independently of the validation expression used in the test. If instead the goal is cell-level inference or a different contrast, the estimand and analysis must be stated accordingly.
Worked Example: Is a permutation budget feasible?
Worked Example: Is a permutation budget feasible?
A hypothetical pipeline takes 2 minutes to run once. The team has an 8-hour compute allocation for one marker test and wants a permutation p-value. Using \(B\) permutations plus the observed data requires approximately \(2(B+1)\) minutes.
An 8-hour allocation is \(8\times60=480\) minutes. The largest integer \(B\) satisfying \(2(B+1)\leq480\) is \(B=239\), because \(2(239+1)=480\). With the common plus-one estimate \[ \hat p_{\mathrm{perm}}=\frac{b+1}{B+1}, \] where \(b\) is the number of permuted statistics at least as extreme as the observed statistic, the smallest attainable value is \(1/(239+1)=1/240\approx0.004167\). For example, if \(b=4\), then \(\hat p_{\mathrm{perm}}=5/240\approx0.02083\).
This budget allows only 239 permutations for one test, before accounting for multiple genes, tuning, or diagnostics. If the pipeline must be rerun separately for many genes, the actual budget may be inadequate. The calculation also says nothing about validity: the permutation must still respect exchangeability under the null, and every adaptive step relevant to the test must be rerun.
Match the strategy to constraints
A useful decision is made in the following order. First specify the inferential target: for example, a marker contrast in the observed dataset, reproducibility in new donors, or evidence against a null of no association under a defined permutation scheme. Then identify the independent sampling units and the steps that used the candidate expression data. Finally assess whether a method can handle those steps at acceptable precision and cost.
If a separate cohort is available and the claim concerns replication, prioritize independent validation. If only one cohort is available, consider a donor-level split when enough independent donors remain in both parts.
Confirm that a clustering or label-assignment rule can be learned in discovery and applied to test observations without adapting it to the test marker outcome.
Use selective inference only when the selection event, assumptions, and calculation cover the clustering and marker-selection procedure actually used.
Specify the null and exchangeable units, rerun the full adaptive pipeline, and estimate the number of runs required for useful tail resolution and the planned multiplicity correction.
Report limitations such as low numbers of donors, distribution shift, an approximate selective model, or limited permutation resolution. No strategy removes the need to define the target and the scope of the claim.
The comparison is not a universal ranking. Independent validation is compelling for external reproducibility but can be unavailable or poorly matched. Data splitting is transparent but sacrifices information and can be unstable when clusters are rare. Selective inference may use the observed data efficiently but can be technically demanding and sensitive to model specification. Full-pipeline permutation can accommodate complicated adaptive procedures when the null permits valid permutations, but computation and exchangeability can be decisive obstacles.
Diagnostics, failure modes, and a common mistake
Before interpreting a result, inspect whether the design and computation match the chosen strategy. For a split, verify that no test observations influenced clustering, marker selection, or tuning, and check whether the held-out groups are represented adequately. For external validation, compare cohort characteristics and measurement conditions, and distinguish a failed transfer from a failed marker. For selective inference, verify that the specified selection event matches the actual pipeline. For permutation, check that permuted units and restrictions reflect the design, and that the pipeline reruns every adaptive step relevant to the statistic.
Power and stability are also diagnostic concerns, but they are not validity guarantees. A stable cluster solution can still yield invalid ordinary post-clustering p-values. Conversely, a valid procedure can be uninformative if the split leaves too few independent samples or the permutation budget yields coarse tail probabilities. The earlier material on cluster stability and marker-test validity makes this distinction especially important.
Common mistake: “There are thousands of cells, so a random cell-level split gives independent validation and enough power.” This reasoning confuses the number of cells with the number of independent biological units. If cells from each donor occur in both partitions, donor-level dependence can connect discovery and test data. The correction is to split at the independent unit relevant to the claim, often the donor, and then assess whether the number of donors is sufficient. If it is not, state that limitation rather than treating the cell count as a substitute.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study has 20 donors and many cells per donor. The scientific claim concerns replication in new donors. Which strategy is most directly aligned with that claim, and what must be checked?
- Why can a data split lose substantial power even when the total cell count is large?
- A team permutes marker values after fixing the observed clusters. Has it performed a full-pipeline permutation? Explain.
- A pipeline takes 3 minutes per run and the budget is 6 hours. What is the largest number \(B\) of permutations possible if the observed run is included, and what is the minimum plus-one permutation p-value?
- Why does a valid selective-inference calculation not automatically apply to a different clustering or marker-selection pipeline?
Solutions. (1) Independent validation is directly aligned with replication in new donors. Check that validation donors are genuinely separate, that the measurement and population are sufficiently comparable, and that group assignment is defined without using the validation marker result. (2) Splitting reduces the number of independent units available for both discovery and testing; many cells within a donor do not create many independent donors. Rare clusters can also become poorly represented in one partition. (3) No. If the observed clusters are held fixed, the permutation does not reproduce the clustering selection step. A full-pipeline procedure reruns relevant adaptive steps under a null-consistent permutation scheme. (4) Six hours is \(360\) minutes, so \(3(B+1)\leq360\), giving \(B\leq119\). Thus \(B=119\), and the minimum plus-one value is \(1/(119+1)=1/120\approx0.008333\). (5) Selective inference depends on a selection event and model that describe the actual procedure. Changing clustering or marker selection can change that event and its conditional distribution, so the original calculation may no longer be calibrated.