Why Held-Out Cells Need a Fixed Assignment Rule
Sample splitting separates the data used to discover clusters from the data used to evaluate marker differences. That separation only has a clear meaning if the held-out cells are given labels by applying a rule learned from the training data. If the held-out subset is clustered anew, its labels are discovered from the very data meant to provide an independent evaluation. The resulting comparison is no longer a test of markers for the training-defined clusters.
A cluster is not automatically a transferable label. A training procedure may produce groups with particular centers, features, and names. To label a new cell, the analyst must specify how its measurements are compared with those training results. The rule, including preprocessing and any thresholds, should be chosen using training data and fixed before evaluating held-out marker expression.
Write \(D_{\mathrm{train}}\) for the training data, \(Z_i\) for the assignment features of held-out cell \(i\), and \(Y_i\) for its expression of a gene being evaluated as a marker. A rule learned from training data can be written \(g_{D_{\mathrm{train}}}(Z_i)\). The output is a training-defined label, such as cluster 1 or cluster 2. This notation also makes an important qualification visible: if \(Y_i\) is itself included in \(Z_i\), the assigned label depends on the tested expression.
A Frozen Nearest-Centroid Rule
One simple rule assigns a cell to the closest training-cluster center in a specified feature space. Suppose the training analysis has produced centers \(\mu_1,\ldots,\mu_K\). For a held-out feature vector \(z\), Euclidean nearest-centroid assignment is
The labels \(1,\ldots,K\) refer to the training clusters and must retain those meanings. A tie rule should be specified, for example assigning tied cells to the lowest-numbered cluster or marking them unassigned. If the training analysis standardizes features, projects them into a lower-dimensional space, or selects features, those operations must also be frozen: apply the training means, scales, loadings, and feature list to held-out cells rather than estimating new ones there.
Worked Example: Assigning a Cell to the Closest Training Center
Suppose training data define two clusters in a two-feature space. Their training centers are \(\mu_1=(0,0)\) and \(\mu_2=(4,0)\). A held-out cell has feature vector \(z=(3,1)\). Assume the features have already been transformed using training-fitted preprocessing, and the same two features are used for both centers and the held-out cell.
The squared distance to cluster 1 is
The squared distance to cluster 2 is
Since \(2<10\), the fixed rule assigns this cell to training cluster 2. Squared distances give the same nearest center as ordinary Euclidean distances, so taking square roots is unnecessary. This assignment says where the cell falls under the specified rule; it does not establish that the cell belongs to a biologically true population.
The rule is only as meaningful as its feature space and distance measure. Features with different scales can dominate Euclidean distance unless scaling is justified and fixed. Correlated features may effectively count the same information more than once. Alternatives include a distance based on a training-estimated covariance matrix or a trained classifier, but whichever rule is chosen must be explicit and applied without refitting on held-out cells.
Worked Example: Applying a Frozen Projection
Suppose training data define a one-dimensional projection \(t=0.6x_1+0.8x_2\), with training cluster centers \(t_1=1\) and \(t_2=5\). The coefficients and centers are fixed. A held-out cell has transformed features \((x_1,x_2)=(3,2)\).
First apply the training projection:
Its absolute distances from the two training centers are \(|3.4-1|=2.4\) and \(|3.4-5|=1.6\). The cell is assigned to training cluster 2. The calculation uses the training projection and centers unchanged. Estimating a new projection from the held-out cells could rotate or otherwise change the feature space, so it would no longer be this fixed assignment rule.
Assignment, Marker Comparisons, and the Split
Assigning held-out cells does not by itself make every marker comparison valid. A central condition is that the assignment rule must not use the held-out values of the gene being tested in a way that creates a difference between the assigned groups. For a straightforward split-based comparison, the training data determine the clusters and rule, while the held-out cells are evaluated using assignment features that do not include the marker under evaluation. The precise null hypothesis and assumptions still matter; a split does not remove confounding, dependence among cells, or every source of bias.
A useful way to see the logic is to condition on the training data. Once \(D_{\mathrm{train}}\) is fixed, the training centers and assignment function are fixed too. If the held-out assignment features do not use marker expression, the labels are determined without selecting cells according to that marker. Under a null model in which the marker’s distribution is the same across the assigned groups (and under the applicable sampling and independence assumptions), the group comparison can then be calibrated for that null. If the assignment uses the tested gene, this argument does not apply: the labels can be functions of the same expression values being compared.
Worked Example: Comparing a Held-Out Gene After Assignment
In a hypothetical example, a frozen rule assigns four held-out cells using a feature that is not the marker gene \(Y\). Two cells are assigned to training cluster 1 and have \(Y\) counts 2 and 4. Two are assigned to training cluster 2 and have \(Y\) counts 7 and 10. The sample mean expression in each assigned group is
The observed difference, cluster 2 minus cluster 1, is \(8.5-3=5.5\) counts per cell. This is a descriptive difference for cells assigned by the frozen rule. Whether a formal test or interval is appropriate requires additional assumptions, including a suitable sampling model and treatment of any dependence among cells from the same donor. If \(Y\) had been used to make these assignments, the displayed difference could partly reflect that selection rather than a held-out marker contrast.
In single-cell studies, cells from one donor are often not independent experimental replicates. A cell-level split can keep training and held-out cells separate computationally while still placing cells from the same donor in both subsets. The assignment rule then remains fixed, but the resulting cell-level comparison may not justify treating every cell as an independent replicate. The split unit and the inferential target must match the study design.
Why Reclustering the Held-Out Subset Is Different
Reclustering means estimating groups from the held-out expression data rather than applying the training-defined assignment function. The held-out data now determine both the labels and the marker values compared across those labels. If the clustering procedure separates cells using expression patterns that include a candidate marker, the procedure can create groups that differ on that marker even when the intended held-out comparison should not have been selected using it. This is another form of data reuse.
Reclustering also changes the object being evaluated. Training cluster 1 might split into two held-out groups, merge with another group, or disappear. Conversely, the held-out clustering might produce groups that have no clear counterpart in training. A label such as “cluster 2” has no stable meaning across separate runs unless a mapping is defined—and matching labels after reclustering does not restore the original fixed assignment procedure.
Worked Example: A New Split Is Not a Training-Cluster Assignment
Suppose the training clusters have marker-independent centers at \(0\) and \(10\) on an assignment feature, and the frozen nearest-center boundary is their midpoint, \(5\). Three held-out cells have feature values \(4.8\), \(5.1\), and \(5.2\). Under the fixed rule, \(4.8\) is assigned to training cluster 1, while \(5.1\) and \(5.2\) are assigned to training cluster 2.
Now consider applying a two-cluster algorithm anew to just these three held-out values. Its partition and centers are determined by those held-out values; they are not the training centers \(0\) and \(10\), and the resulting labels are not automatically training cluster 1 and training cluster 2. The reclustering answers a different question—how this subset is partitioned by that algorithm—rather than how each cell maps to the training clusters. Even if the two procedures happen to yield labels that look similar here, they are not interchangeable procedures.
Diagnostics and Failure Modes
A fixed assignment can still be unreliable. Before interpreting assigned-group marker comparisons, inspect how the rule behaves on held-out cells and document any exclusions or ambiguous assignments. Useful checks include:
- Distance from the assigned center. Many large distances can indicate that held-out cells are unlike the training cells. A nearest-center rule always returns a label unless an explicit rejection rule is added.
- Assignment margins. If the two smallest distances are nearly equal, a small measurement change could alter the label. Report such ambiguity or use a pre-specified tie or abstention rule.
- Cluster representation. A training cluster receiving no held-out assignments cannot support a held-out comparison. Very small assigned groups can also make estimates unstable.
- Preprocessing consistency. Confirm that held-out cells used the training feature list, scales, and projection. Refitting these on the held-out subset changes the rule.
- Feature leakage. Check whether the tested marker, or a feature derived from it, contributed to assignment. If so, a simple split does not protect that marker comparison from reuse.
One possible safeguard is an abstention rule: assign a cell only if its closest center is within a pre-specified maximum distance, and otherwise mark it unassigned. The threshold must be set from training data or specified in advance, not chosen after inspecting which held-out assignments produce a favorable marker result. Abstention changes the population to which the comparison applies: it concerns assigned cells meeting the rule, not every held-out cell.
Worked Example: Using a Pre-Specified Distance Threshold
Suppose a training-based rule assigns a held-out cell to its closest center only when the Euclidean distance is at most 2.5; otherwise it marks the cell unassigned. The rule and threshold were fixed using training data. A cell has distances 2.0 and 3.1 from the two training centers. Its nearest-center distance is \(2.0\), and \(2.0\leq2.5\), so it is assigned to the closer training cluster. A second cell has distances 2.7 and 3.4. Although its nearest center is identifiable, \(2.7>2.5\), so it is unassigned. Assigning the second cell anyway would contradict the specified rule.
Choosing Among Assignment Approaches
| Approach | What is fixed from training? | Main caution |
|---|---|---|
| Nearest centroid | Feature space, preprocessing, cluster centers, and distance rule | May assign outlying cells unless rejection is allowed |
| Trained classifier | Features, fitted parameters, and prediction rule | Prediction is not independent of any tested gene used as an input |
| Recluster held-out cells | Usually nothing defining the training partition | Labels are newly estimated from test data and need not correspond to training clusters |
| Manual or post hoc matching | Matching may depend on observed held-out patterns | Can reintroduce data-dependent choices; requires a defined mapping procedure |
No assignment method is universally best. Nearest-centroid rules are transparent but can be geometrically simplistic; classifiers can capture more complex boundaries but may be harder to interpret and still depend on their inputs. Reclustering may be useful for a separate exploratory question, but it does not serve as a fixed mapping to training clusters for an honest held-out comparison.
Common Mistake and Practical Summary
Wrong reasoning: “The held-out cells are separate from the training cells, so I can cluster them again and still call their marker comparison independent.” The error is treating separation of rows as sufficient. Reclustering uses held-out expression to define the groups being compared, so the labels are no longer fixed by training alone.
Correction: Fit the cluster definition and assignment rule using training data, freeze every relevant step, and apply that rule to held-out cells. Then check whether assignment features overlap with the marker being evaluated, whether cells are ambiguous or outlying, and whether the sampling design supports the intended inference. If the held-out cells are reclustered, describe that as a new exploratory clustering rather than a test of markers for the original training clusters.
Check Your Understanding
Answer each question, then compare with the solutions.
- Training centers are 1 and 7 on a one-dimensional feature. A held-out cell has value 5. Which training cluster receives it under nearest-centroid assignment?
- Why does estimating a new feature scale from held-out cells alter a frozen assignment rule?
- A marker gene is included among the features used by a classifier that assigns held-out cells. Does sample splitting alone establish an honest comparison for that marker? Explain.
- What does an abstention threshold change about the interpretation of a held-out marker comparison?
Solutions. (1) The distances are \(|5-1|=4\) and \(|5-7|=2\), so the cell is assigned to training cluster 2. (2) The transformation applied to the held-out cell would be estimated using the held-out subset rather than the training-fitted scale; the resulting feature coordinates and assignments may change. (3) No. The held-out label depends on the marker expression, so the split alone does not prevent reuse of that expression in defining the groups. (4) The comparison applies to cells that meet the assignment threshold, not automatically to all held-out cells; unassigned cells are outside that comparison.