Holding Out Genes Used For Clustering

Why Hold Out Genes?

Clustering and marker testing can use the same expression measurements in a circular way: a clustering algorithm may form groups partly because certain genes differ, and tests on those same genes can then overstate the evidence for group differences. Feature splitting offers a different way to reduce this double dipping. Instead of dividing cells into training and testing sets, divide genes into two disjoint sets: use one set to define clusters and the other to test for markers.

This changes which measurements are reused, but it does not make the analysis automatically confirmatory. The validity of a test on a held-out gene depends on how that gene relates to the genes used for clustering, how the data were preprocessed, and whether the cell-level testing model is appropriate. In particular, genes that are not identical can still be strongly dependent.

The split should be made before examining which held-out genes appear promising as markers. For example, one can randomly assign genes to \(A\) or \(B\), or allocate genes by a prespecified rule. If genes are selected for \(A\) after screening all genes for patterns that appear useful for clustering, that screening step may itself use information from the proposed testing set. The entire adaptive procedure—not merely the final clustering call—matters.

When Does the Split Justify a Marker Test?

Suppose the clustering procedure is a fixed function of \(X_A\), producing labels \(C_i=f(X_A)_i\). Once \(X_A\) is observed, those labels are fixed for the purpose of a held-out-gene test. For a gene \(g \in B\), consider a null hypothesis that its expression distribution is the same across the groups being compared. A conventional two-group test can have its usual calibration conditional on the clustering data if the held-out expression values are independent of the clustering data and the test's own assumptions hold.

For a simple sufficient condition, suppose the held-out gene's measurements are independent of \(X_A\), and under the null they have the same distribution across cells. Conditional on \(X_A\), the cluster assignments are then fixed, while the held-out measurements retain the same null distribution in both assigned groups. A valid test applied to those groups can therefore have its stated type I error rate, conditional on the clustering data. Independence is sufficient, not necessary; more general conditional models may also justify inference, but they must be specified and supported.

This statement concerns the held-out gene's relationship to the clustering measurements under the null; it does not assert that genes are generally independent in single-cell data. It also does not replace the assumptions of the chosen marker test. For example, a Welch two-sample t-test assumes independent observations within each group and uses an approximate reference distribution; with small samples, normally distributed measurements support its usual approximation. If cells from the same donor are correlated, treating every cell as independent can understate uncertainty even when the gene split is otherwise well designed.

A Conditional-Validity Justification

The logic can be made explicit. Let \(C=f(X_A)\) denote all cluster labels and suppose they define two groups with sizes \(n_1\) and \(n_2\). If the held-out expression vector \(Y_g\) is independent of \(X_A\), then it is also independent of any function of \(X_A\), including \(C\). Under the null that the expression measurements are identically distributed across cells, conditioning on \(X_A\) fixes which cell indices belong to each group but does not change the distribution of their held-out measurements.

for every measurable set \(D\). Thus, conditional on the clustering data, the test compares two groups selected by fixed indices, and the null distribution used by a valid test remains appropriate. This is a conditional argument: it explains why independent held-out features can avoid the specific selection effect created by clustering on the tested gene. It does not prove validity when the test model is wrong, observations are dependent, or the held-out gene is associated with the clustering data.

The argument also clarifies the difference from holding out cells. Holding out cells separates the observations used to learn a clustering rule from those used to assess it. Holding out genes separates the features used to construct labels from the features tested as markers, while often retaining the same cells for both steps. The two strategies address different sources of reuse and have different assumptions.

A Worked Marker Test on Held-Out Expression

For a numerical illustration, consider hypothetical, normalized log-expression values for one gene in \(B\). Assume the two clusters contain independent cells, and suppose the values are approximately normal within each group. A two-sided Welch test compares the group means without requiring equal population variances. These are assumptions for this example, not general properties of single-cell measurements.

Why Gene Dependence Weakens the Guarantee

Genes often share biological regulation, cell-state signals, or technical influences. If a held-out gene is correlated with genes in \(A\), cluster labels based on \(A\) can also be associated with that held-out gene. The held-out gene did not directly define the labels, but its correlated expression can carry information about the same structure. In that case, the conditional-independence step in the justification does not follow merely from \(A \cap B = \varnothing\).

Preprocessing can create or amplify dependence as well. For instance, if a normalization factor is computed using all genes and then applied to both \(A\) and \(B\), the transformed value of a gene in \(B\) may depend on measurements from \(A\). Feature selection or dimensionality reduction performed using all genes before the split can likewise allow information from \(B\) to affect clustering. A defensible split must account for the full pipeline, not only which columns are passed into the final clustering function.

Diagnostics, Alternatives, and Limitations

A feature split is most credible when the split is prespecified, the clustering pipeline uses only \(A\), and the marker pipeline uses only \(B\). Useful checks include reviewing the full preprocessing code, checking whether size factors or selected features use both sets, and examining gene dependence or shared latent structure. A large correlation between genes across the split is a warning that the simple independence justification may not apply; a small observed correlation does not prove conditional independence.

The main trade-off is that fewer genes are available for each purpose. Restricting clustering to \(A\) can make clusters less stable or less biologically informative, while testing only \(B\) reduces the number of candidate markers. Repeating the split and reporting only favorable results reintroduces selection unless the repetitions and selection rule are accounted for. A feature split also does not by itself address multiple testing across many held-out genes, donor-level dependence, batch effects, or misspecification of the marker test.

Compared with holding out cells, feature splitting leaves the same cells in the clustering and testing steps but separates the gene measurements used for each. Compared with validating markers in an independent dataset, it does not provide a new sample of cells or donors. These approaches therefore support different kinds of evidence; none should be described as a universal substitute for the others.

• Warning sign: clustering inputs, selected features, or normalization factors depend on genes intended for testing.
• Warning sign: strong coexpression across the split makes the held-out-gene independence assumption implausible.
• Warning sign: many cells come from a small number of donors, but the test treats every cell as an independent replicate.
• Warning sign: only a favorable random split is reported, with no prespecified rule for handling repeated splits.