Why Separate Discovery From Testing?
A marker gene is often found by searching for expression differences between clusters. But if the same expression measurements both determine the clusters and test the differences, the apparent separation can partly be a consequence of how the groups were chosen. The earlier material on double dipping and the null hypothesis after data-adaptive clustering describes why an ordinary test may then have a p-value that is too small.
Sample splitting gives a direct way to separate those tasks. Use one subset of independent observational units to discover clusters, then reserve another subset for testing a prespecified gene contrast. The essential idea is not merely to divide a table in two: the test data, including the information used to define the test groups, must not have influenced the discovery or the choice of the tested genes.
This definition includes an important practical qualification. Clustering the discovery set does not by itself tell us which held-out cells belong to each discovered cluster. A held-out comparison requires group labels that are available independently—for example, labels from a prespecified external annotation or a design that already identifies the groups. If held-out labels are instead constructed from the held-out expression data, the independence condition may fail. Assigning held-out cells to clusters is a distinct step, not something that sample splitting automatically accomplishes.
Designing a Split That Supports Inference
The split must be made at an appropriate independent unit. If cells from the same donor share biological and technical features, randomly putting some of a donor's cells in discovery and others in testing does not generally make the two sets independent. For inference about a population of donors, assign whole donors to the two sets. Cell-level splitting may be relevant to a narrower prediction question, but it does not turn correlated cells into independent biological replicates.
The discovery analysis should be finalized without consulting test-set expression. This includes the clustering choices, the target contrast, and the list of genes to be tested. A test-set filter that retains genes because they look different in the test data is another data-dependent selection step. Likewise, repeatedly changing the clustering or choosing among splits after inspecting test p-values undermines the separation.
For a particular gene, suppose the test-set groups are fixed independently of that gene's test expression, and the null hypothesis is that its mean expression is equal in the two groups. A two-sample t-test can be used under the assumptions that the independent test units are sampled independently within groups, the test observations are approximately normal within each group (or sample sizes support a suitable approximation), and—if using the pooled t-test—the two group variances are equal. These assumptions concern the test units and outcome being analyzed, not merely the number of cells in the data file.
The justification is the law of total probability: average the conditional rejection probability over all possible discovery outcomes. For each fixed outcome, the hypothesis and test are already determined, and the held-out test retains its stated error control. Averaging probabilities that are each no greater than \(\alpha\) cannot produce a probability greater than \(\alpha\). This argument does not make an invalid test valid: dependence between the splits, data-adaptive test labels, or a misspecified test model can invalidate the conditional guarantee.
Worked Examples
Worked Example: Allocating Donors Before Cluster Discovery
Suppose a hypothetical study has 20 donors, 10 in each of two prespecified experimental conditions. The investigators want to discover clusters in a subset and then test a gene contrast in held-out donors. They randomly allocate 6 donors from each condition to discovery and the remaining 4 from each condition to testing.
The resulting split has \(6+6=12\) discovery donors and \(4+4=8\) test donors. Stratifying by condition ensures both conditions occur in each subset; random allocation within condition prevents investigators from choosing donors based on observed expression. Clustering and marker selection use only the discovery donors.
For the proposed test to be interpretable, the held-out group labels must also be available without using the held-out expression of the tested gene. For example, assume an independent, prespecified annotation supplies the group labels. The test then compares expression between those labeled groups in the 8 held-out donors. If the investigators instead inspect the test expression, alter the clustering, and define labels to maximize the gene contrast, the test set has been reused and the split no longer protects the test.
This allocation does not guarantee that the test will have high power, and it does not fix dependence if several cells from a donor are treated as independent replicates. The donor, rather than each cell, is the independent unit for this example's donor-level inference.
Worked Example: A Two-Sided Held-Out Marker Test
Consider a hypothetical held-out comparison of one prespecified gene between two independently labeled groups. The observations are donor-level pseudobulk expression summaries, in log2-normalized expression units; there are 8 independent donors per group. Assume approximate normality within groups and equal variances. The group means are 4.80 and 3.60, and both sample standard deviations are 1.00251.
The estimated difference, group 1 minus group 2, is \(4.80-3.60=1.20\) log2-normalized expression units. For the pooled t-test, the pooled standard deviation is \(s_p=1.00251\), because the two reported standard deviations are equal. The standard error is
The test statistic for \(H_0:\mu_1-\mu_2=0\) against \(H_a:\mu_1-\mu_2\ne 0\) is
The two-sided p-value is \(2P(T_{14}\geq 2.394)=0.03123\), where \(T_{14}\) has a t distribution with 14 degrees of freedom. At \(\alpha=0.05\), this is evidence against equal mean expression for this prespecified contrast. It is not the probability that the null hypothesis is true, nor does it establish that the gene causes a difference between groups.
A 95% confidence interval uses the critical value \(t_{0.975,14}\approx2.145\):
Under the model assumptions, this interval describes plausible values for the population mean difference in log2-normalized expression units. It excludes zero, consistent with the two-sided test at the 5% level. The interval and p-value address this one prespecified contrast; they do not account for testing many genes.
Worked Example: The Power Cost of Holding Data Out
Suppose, for illustration, that 40 independent test-eligible units would otherwise be available, split equally between two groups, and that each group's expression has standard deviation \(\sigma\). With all 40 units, the standard error of the difference in means is
If half the units are reserved for discovery and only 20 remain for testing, with 10 per group, the test-set standard error becomes
Thus \(SE_{\mathrm{split}}/SE_{\mathrm{all}}=0.4472/0.3162\approx1.414\): the standard error is about 41% larger, for the same underlying difference and variance. This usually reduces power, meaning a greater chance of failing to reject a false null hypothesis. The calculation assumes independent units, equal group sizes, and equal variances; it illustrates the cost of splitting rather than guaranteeing a particular power for a real study.
Diagnostics, Alternatives, and Limitations
Before trusting a held-out result, check the analysis path, not just the final test statistic. Confirm that the test units were not used in clustering or marker selection; that any test labels were assigned independently of the tested expression; and that the split was made at the unit needed for the intended inference. For a t-test, inspect group summaries and residual behavior for strong skewness or outliers, and check whether the equal-variance assumption is plausible if using the pooled version. A small p-value cannot repair a failure of these conditions.
Sample splitting is one approach to the double-dipping problem, not a universal solution. Reusing all observations without accounting for data-driven selection can yield optimistic marker evidence. An independent replication dataset can test a frozen marker claim and offers a separate check on reproducibility, but it requires additional data. Methods that explicitly account for selection may use data more efficiently, but they rely on their own assumptions and are not interchangeable with a simple held-out test.
Several limits are especially important in single-cell analyses:
- Labels are not automatic. A discovery clustering does not, by itself, define test-set memberships. The test needs an independent source of labels or a defensible procedure for obtaining them.
- Cells may not be independent. Treating thousands of cells from a few donors as thousands of independent biological replicates can understate uncertainty. Splitting cells within donors does not solve this problem.
- Splitting costs precision. Fewer test units generally means larger standard errors and lower power. A split that leaves too few units for testing can produce uninformative intervals.
- One valid p-value is not family-wise or false-discovery control. Testing many genes creates a multiple-testing problem even when every individual held-out p-value is valid. The gene family and error criterion must be addressed separately.
- Repeated peeking breaks the design. Trying many splits or analyses and reporting the one with the smallest p-value reintroduces selection based on test outcomes.
The main benefit is a clear separation between a data-adaptive discovery claim and a confirmatory test. The main price is reduced information in the test set. The split, analysis plan, unit of inference, and origin of test labels should therefore be stated explicitly when reporting results.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Why can randomly splitting cells from the same donors fail to support inference about a population of donors?
- A gene is selected because it has the smallest p-value in the held-out set. Is the ordinary p-value still an honest test of that selected gene? Explain.
- In the worked t-test, state the estimated difference, standard error, degrees of freedom, and conclusion at the 5% level.
- Does a valid held-out p-value for one gene control the error rate across 5,000 tested genes?
- Why is a split that reserves more data for discovery not always preferable?
Solutions. (1) Cells from one donor can be correlated, so the two subsets are not independent donor samples; inference should respect donor as the unit. (2) No. Choosing the smallest test-set p-value uses the test results for selection, so its ordinary single-test calibration does not account for that choice. (3) The estimated group 1 minus group 2 difference is 1.20 log2-normalized units, \(SE=0.501255\), and \(df=14\). Since \(p=0.03123<0.05\), reject equality of means for this prespecified contrast under the stated assumptions. (4) No. Individual validity does not provide multiplicity control across thousands of tests. (5) More discovery data may help stabilize clustering, but it leaves fewer test units, increasing standard errors and reducing power for marker tests.