Why Preprocessing Sensitivity Matters
A clustering algorithm does not operate on a cell’s biology in the abstract. It operates on a representation of the measured data: counts may be normalized and transformed, genes may be scaled or filtered, and cells may be represented in a selected lower-dimensional space. These choices can change distances or graph connections and, in turn, the resulting assignments. A cluster that appears only under one defensible preprocessing choice deserves closer scrutiny than one that persists across several reasonable choices.
Preprocessing sensitivity is therefore a validity diagnostic. It asks whether a conclusion about the partition depends strongly on how the data were prepared. It does not identify one universally correct preprocessing pipeline, and it cannot establish that a stable cluster corresponds to a real cell type or state. The aim is to make dependence on analytical choices visible, then interpret that dependence using the scientific question and the properties of the data.
“Defensible” matters. A pipeline should be a plausible analysis for the data and question, not an arbitrary transformation chosen to manufacture disagreement. For example, comparing two reasonable normalization choices may be informative; comparing a valid normalization with an intentionally distorted one is not evidence that ordinary preprocessing uncertainty is consequential.
What to Compare and What to Hold Fixed
Define each candidate pipeline before examining its cluster results. A pipeline might specify a normalization method, whether and how features are scaled, and the representation used for clustering. If the purpose is to assess one decision, change that decision while holding the others fixed. If the purpose is to assess the robustness of a final conclusion to plausible workflow alternatives, compare a small set of complete, defensible pipelines as well. These are complementary questions: the first helps locate a source of sensitivity; the second assesses sensitivity to realistic combinations of choices.
For a clean comparison, use the same cells and the same clustering settings across the pipelines being compared. Keep the feature set fixed when testing a normalization or scaling choice, unless changing the feature set is itself the decision under study. Likewise, when testing a dimensional representation, avoid simultaneously changing normalization and scaling. If several choices change at once, a difference in the partition may be real but its source is ambiguous.
Compare partitions, not numeric cluster labels. Labels such as 1 and 2 are arbitrary: a relabelling does not change the underlying grouping. For the same set of cells, the adjusted Rand index (ARI) is one useful summary of agreement. Let \(n_{ij}\) be the number of cells assigned to cluster \(i\) in the first partition and cluster \(j\) in the second; let \(a_i=\sum_j n_{ij}\), \(b_j=\sum_i n_{ij}\), and \(n\) be the number of cells compared. For an integer \(m\), \(\binom{m}{2}=m(m-1)/2\) counts pairs among \(m\) items.
The ARI is 1 for identical partitions when its formula is defined; it is near 0 when agreement is roughly what its chance adjustment expects, and it can be negative. Its denominator is the standard ARI normalization, not necessarily a maximum agreement attainable for arbitrary fixed cluster sizes. The index summarizes pairwise agreement, but it does not say which cells changed, which choice caused the change, or whether either partition is biologically valid.
Worked Example: Computing Agreement for Two Partitions
Suppose six hypothetical cells receive these assignments under two preprocessing pipelines. Pipeline A groups cells as \(\{1,2,3\}\) and \(\{4,5,6\}\). Pipeline B groups them as \(\{1,2,4\}\) and \(\{3,5,6\}\). Each partition has two clusters of three cells. The contingency table is:
| B: cluster 1 | B: cluster 2 | Row total | |
|---|---|---|---|
| A: cluster 1 | 2 | 1 | 3 |
| A: cluster 2 | 1 | 2 | 3 |
| Column total | 3 | 3 | 6 |
There are \(\binom{2}{2}+\binom{1}{2}+\binom{1}{2}+\binom{2}{2}=2\) cell pairs placed together by both partitions. Each partition has \(\binom{3}{2}+\binom{3}{2}=6\) within-cluster pairs, and there are \(\binom{6}{2}=15\) pairs in total. Substitution gives:
The partitions agree on fewer jointly grouped pairs than expected under the ARI’s chance adjustment, so this small example has a negative ARI. The result describes disagreement between these assignments; it is not a p-value and does not establish that either preprocessing pipeline is wrong.
Designing and Interpreting a Sensitivity Analysis
A useful analysis starts with a specific question. “Does clustering change?” is too broad if the analyst has not specified which choices are uncertain and which features of the result matter. Consider, for example, whether a putative group persists under two reasonable normalization methods. Keep the cell set, downstream clustering settings, and other preprocessing steps the same. Compare the resulting partitions, then inspect the cluster sizes and relevant biological summaries. If assignments differ, ask whether the change is concentrated at a boundary, involves a small group, or alters the main interpretation.
When choices are changed jointly, compare a small, prespecified collection of realistic pipelines rather than an unrestricted grid of every possible option. A broad search can produce many opportunities to find a preferred result. Report the choices considered and the comparison being made so that sensitivity is not hidden by presenting just one pipeline. A high ARI alone is not enough: two partitions could agree globally while differing in a scientifically important small group. Conversely, a lower ARI does not explain whether the change is meaningful. Cluster sizes, marker summaries, and the scientific question help provide context.
Name the normalization, scaling, or dimensional-representation choice whose effect you want to assess, and state which choices remain fixed.
Use options that are plausible for the data and question. Avoid treating arbitrary or deliberately poor transformations as equally reasonable alternatives.
Use a common cell set and consistent clustering settings for an interpretable comparison. If cell sets differ, the direct partition comparison needs additional care and is no longer a simple like-for-like assessment.
Use a partition-comparison measure such as ARI, and also inspect cluster sizes and biological summaries. Ask whether the interpretation, not just the label numbering, changes.
State which choices were compared and what changed. Treat stability across those choices as evidence of robustness to the choices tested, not proof of truth.
Worked Example: Isolating a Normalization Choice
In a hypothetical analysis of 2,000 cells, an analyst wants to test whether a proposed two-group partition depends on the normalization method. The feature set, scaling, dimensional representation, cell set, and clustering settings are held fixed. Only normalization changes. The resulting ARI is 0.91.
The high ARI indicates strong overall agreement between the two partitions, but it does not mean that exactly 91% of cells kept their assignments. The ARI is a chance-adjusted pairwise measure, not a cell-level retention percentage. The analyst therefore also checks the cluster sizes and the identities of cells whose assignments differ. Suppose the groups remain similar in size and the small set of changed cells lies near the same transition in the marker summaries. In that hypothetical case, the broad two-group interpretation appears relatively insensitive to this particular normalization comparison, while exact cell assignments are not identical.
This conclusion is deliberately limited: it concerns the two normalization choices tested under the fixed settings. It does not establish robustness to different scaling or dimensional-representation choices, nor does it validate the biological interpretation of the groups.
Worked Example: Comparing Several Defensible Pipelines
Suppose a hypothetical study has 5,000 cells. The analyst compares four prespecified pipelines on the same cells and with the same clustering settings. The pipelines differ in normalization, scaling, or dimensional representation as listed below. For illustration, each ARI is calculated against Pipeline A.
| Pipeline | Choice relative to A | ARI versus A | Hypothetical observation |
|---|---|---|---|
| A | Reference pipeline | 1.000 | Two main groups and one smaller group |
| B | Alternative normalization only | 0.940 | Similar group sizes; a few boundary assignments differ |
| C | Alternative scaling only | 0.620 | The smaller group is split and several assignments change |
| D | Alternative dimensional representation only | 0.880 | Main groups remain, but the smaller group partly merges |
Pipeline B agrees strongly overall with A; the conclusion should still be phrased as agreement under this comparison, not as a claim that 94% of cells are unchanged. Pipeline C shows substantially less agreement, so scaling is a candidate source of sensitivity in this analysis. Pipeline D also changes the smaller group’s status. Those observations motivate closer examination of the affected cells and biological summaries, rather than an automatic decision to choose the pipeline with the highest ARI.
Because each comparison changes one listed choice relative to A, it helps identify which choices coincide with changes in these results. It does not show that a choice alone caused every affected assignment in a general sense, nor does it assess all possible interactions among choices. A follow-up comparison of a few plausible complete pipelines can determine whether the combined alternatives lead to similar concerns.
Diagnostics, Alternatives, and Common Pitfalls
A sensitivity analysis is informative only if the comparison itself is well controlled. Check that the same cells are being compared, that cluster labels are treated as arbitrary, and that one intended preprocessing choice—not several unnoticed changes—distinguishes the pipelines. Confirm that every alternative is defensible for the scientific question. If a result changes, inspect cluster sizes and relevant biological summaries rather than relying on one global index.
Preprocessing sensitivity complements, but is not interchangeable with, other robustness checks. The earlier material on Feature Perturbations and Cluster Robustness considers changes to features; preprocessing sensitivity instead changes how measured data are prepared or represented. Cell subsampling asks whether conclusions persist when the observed cells change. These procedures probe different sources of variation. Agreement across one type of comparison does not guarantee agreement across another.
Internal validity summaries such as silhouette width or graph conductance answer different questions: they assess properties of a partition or graph under a chosen representation, rather than directly measuring agreement between preprocessing alternatives. A partition can score well under one representation and still change substantially when that representation changes. Conversely, agreement across pipelines does not guarantee strong separation or biological validity.
- Pitfall: treating ARI as a percentage of unchanged cells. An ARI of 0.90 is not a statement that 90% of cells retain their assignments. It measures chance-adjusted pair agreement; inspect changed cells directly if cell-level retention matters.
- Pitfall: changing several steps and attributing the result to one. If normalization, scaling, and representation all change at once, the comparison cannot isolate their individual contributions. Change one at a time for diagnosis, or describe the result as a comparison of complete pipelines.
- Pitfall: calling stability proof of biological truth. A group can persist across the choices tested and still reflect a technical or analytical artifact. Robustness is evidence about dependence on tested choices, not a guarantee of biological interpretation.
- Pitfall: interpreting a global score without inspecting small groups. A high overall ARI can coexist with a consequential change in a small cluster. Examine cluster sizes and the scientific role of affected cells.
The main limitation is scope: no finite set of pipelines explores every reasonable preprocessing choice, and the ARI compresses a complex comparison into one number. Results also depend on the chosen cells, features, and clustering settings. Report sensitivity as conditional on the choices actually tested. Do not present a preferred pipeline as objectively correct solely because it agrees with another pipeline or yields a more appealing partition.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why should the same cells be used when directly comparing two partitions with ARI?
- Two pipelines have ARI 0.88. Is it correct to conclude that 88% of cells kept their assignments? Explain.
- An analyst changes normalization and scaling together and observes a different partition. What can and cannot be concluded?
- A high ARI is reported, but a rare group merges into a larger group under one alternative pipeline. Why should the analyst inspect more than the ARI?
- How does preprocessing sensitivity differ from cell subsampling as a robustness check?
Solutions. (1) The direct comparison should refer to the same set of units; otherwise, disagreement can reflect different cells as well as different assignments. (2) No. ARI is a chance-adjusted pairwise agreement measure, not the proportion of cells retaining labels. (3) The complete change of pipeline is associated with a different partition, but the comparison cannot identify whether normalization, scaling, or their combination is responsible. (4) A global index can conceal an important change in a small group; inspect cluster sizes, affected cells, and relevant biological summaries. (5) Preprocessing sensitivity changes how the observed data are prepared or represented, whereas cell subsampling changes which cells are included. They address different sources of instability.