Why Perturb the Features?
A clustering can depend on which genes represent each cell. A small change to the feature set may alter distances, the low-dimensional representation, or the graph used for clustering—even when the cells and all other analysis choices remain fixed. Feature perturbation asks a focused question: does the partition persist under plausible, modest changes to the genes supplied to the analysis?
This is a conditional robustness check, not a test of whether the clusters are biologically true. It complements cell subsampling, which changes which cells are analyzed. Here, retain the same cells in every run and vary only the features according to a defined perturbation scheme. The introductory material on cluster stability motivates comparing repeated runs; the key design issue here is to make the feature change interpretable.
For example, one can repeatedly select a fixed-size subset from a prespecified pool of eligible genes, or omit a modest fraction of genes from the baseline set. The clustering must be rerun for each feature set: recompute any representation that depends on the selected genes, then apply the same dimensionality, graph, clustering, and resolution choices. Otherwise, the comparison may not reflect the consequences of changing features.
Designing a Controlled Perturbation
Specify the perturbation before inspecting the resulting stability scores. State the eligible gene pool, the number or proportion of features changed, how perturbed sets are generated, and how many runs are made. A useful design uses perturbations that are modest enough to represent plausible feature uncertainty but large enough to meaningfully test sensitivity. There is no universal perturbation size: it should be justified in the context of the feature-selection procedure and scientific question.
Keep fixed the cell set, normalization and transformation, number of dimensions, graph-construction settings, clustering algorithm and resolution, and other analysis choices. Recalculate only quantities that necessarily depend on the changed gene set, such as the feature-based representation. A fixed random seed can help keep the comparison focused on feature changes, although a fixed seed does not establish that the procedure is generally insensitive to its own randomness.
Features are not generally independent replicates. Genes can be correlated, and several genes may represent the same pathway or cell program. Consequently, randomly omitting individual genes may test sensitivity to individual feature identities without adequately testing dependence on a whole correlated module. When scientifically motivated, consider a separate, prespecified perturbation that omits or downweights a modest gene block. Report such a block perturbation distinctly from random gene-level resampling.
Comparing Partitions
To compare two runs on the same cells, use a measure that does not depend on the arbitrary names assigned to clusters. The adjusted Rand index (ARI) compares pairs of cells: it rewards agreement about whether pairs are together or apart, while adjusting for the agreement expected under a chance model with the observed cluster-size margins fixed.
Suppose one partition has clusters \(U_1,\ldots,U_R\), and another has clusters \(V_1,\ldots,V_S\). Let \(n_{ij}\) be the number of cells in both \(U_i\) and \(V_j\), with row totals \(a_i\), column totals \(b_j\), and \(n\) cells in total. For any integer \(m\), \(\binom{m}{2}=m(m-1)/2\) is the number of unordered pairs among \(m\) cells. The ARI is
An ARI of 1 indicates identical partitions; values near 0 indicate agreement around the chance-adjusted reference, and negative values indicate less agreement than that reference. The index is unchanged if cluster labels are renamed. It is a global summary, not the percentage of cells that kept their cluster, and it does not identify which particular cluster changed.
Worked Example: Calculating Agreement for Eight Cells
Hypothetical data. A baseline run assigns cells 1–4 to cluster A and cells 5–8 to cluster B. A perturbed-feature run assigns cells 1, 2, 3, and 5 to X, and cells 4, 6, 7, and 8 to Y. The contingency table is:
| X | Y | Row total | |
|---|---|---|---|
| A | 3 | 1 | 4 |
| B | 1 | 3 | 4 |
| Column total | 4 | 4 | 8 |
The sum of within-cell pair counts is \(3+0+0+3=6\), since the four cells in the table have sizes 3, 1, 1, and 3. Each partition has two clusters of size 4, so each margin contributes \(\binom{4}{2}+\binom{4}{2}=12\). The total number of cell pairs is \(\binom{8}{2}=28\). Thus the chance-adjustment term is \(12(12)/28=5.1429\), and
The partitions have limited agreement beyond the chance-adjusted reference. The ARI of 0.1250 does not mean that 12.5% of the cells retained their assignments; it is a pair-based, adjusted partition-agreement measure.
Why the Adjustment Has This Form
The first term in the numerator, \(\sum_{i,j}\binom{n_{ij}}{2}\), counts pairs assigned together in both partitions. If cluster sizes are fixed and the two partitions are arranged at random relative to one another, the expected value of this count is
To see why, take any pair of cells. The proportion of all pairs that fall together in the first partition is \(\sum_i\binom{a_i}{2}/\binom{n}{2}\); in the second it is \(\sum_j\binom{b_j}{2}/\binom{n}{2}\). Under the fixed-margin chance arrangement, the expected number together in both is \(\binom{n}{2}\) times the product of these proportions, giving the expression above. Subtracting that expected count centers the observed agreement at the chance reference. The denominator scales this centered agreement by the average of the two partitions’ total within-cluster pair counts, also centered by the same expected count. This gives the adjusted index, provided the denominator is nonzero.
The chance reference is a mathematical comparison, not a scientific null hypothesis about whether cell types exist. In particular, an ARI does not supply a p-value or a confidence interval for biological validity.
Summarizing Repeated Perturbations
Compare every perturbed run with the baseline partition, using the same cells in each comparison. Report the individual ARIs as well as a summary such as their median and range. A mean alone can conceal a small number of feature sets that cause substantial changes. If a stability threshold is useful for a decision, choose it in advance and describe it as a practical rule, not a universal statistical cutoff.
Worked Example: A Distribution of Feature-Stability Scores
Hypothetical data. An analyst creates six feature sets, each replacing 20% of the baseline genes with genes sampled from a prespecified eligible pool. All six runs include the same cells and use the same fixed analysis choices. Their ARIs against the baseline are 0.91, 0.88, 0.86, 0.89, 0.42, and 0.90.
In ascending order, the values are 0.42, 0.86, 0.88, 0.89, 0.90, and 0.91. The median is \((0.88+0.89)/2=0.885\), and the range is 0.42 to 0.91. If the analyst had prespecified 0.80 as a practical flag, five of six runs meet that rule. The low value remains important: a high median does not show that every modest feature change leaves the result intact.
The appropriate conclusion is that most tested feature sets produce strong global agreement with the baseline, but at least one tested set produces much lower agreement. The scores alone do not reveal which genes caused the change or whether the changed partition is less biologically meaningful.
Diagnosing Sensitivity and Choosing Comparisons
When a perturbation produces low agreement, first verify that the comparison is controlled: the cells are identical, the intended feature change was applied, and other settings were not inadvertently altered. Then examine whether the change is reproducible across independently generated feature sets of the same kind. A single low score may reflect an unusual draw, whereas a consistent decline under a defined perturbation suggests a more systematic sensitivity.
Compare perturbations that answer different questions separately. Randomly replacing individual genes probes sensitivity to feature identities in the eligible pool. Omitting a prespecified gene block probes dependence on that block. Varying the number of selected genes probes sensitivity to feature-set size. Do not combine all these changes in one run and attribute the resulting difference to a single cause.
Worked Example: Testing Dependence on a Gene Module
Hypothetical data. A baseline analysis uses 120 genes. In addition to ordinary random replacements, an analyst tests whether a 12-gene module is influential. Across five modest random replacements of 12 genes, the ARIs are 0.88, 0.90, 0.89, 0.87, and 0.91. Their mean is \((0.88+0.90+0.89+0.87+0.91)/5=0.890\). When the analyst instead omits the prespecified 12-gene module, the ARI is 0.48.
This comparison suggests that the result is not equally sensitive to every 12-gene change: it is more sensitive to omission of the specified module than to the tested random replacements. The interpretation is conditional on the perturbation design. It does not show that the module is spurious or that clusters found without it are correct. A useful follow-up is to report the module perturbation separately and check that the result is reproducible under the same controlled omission.
Feature perturbation differs from cell subsampling because it holds the analyzed cells fixed; cell subsampling instead tests sensitivity to which cells are included. It also differs from changing normalization or other preprocessing choices, which changes a different part of the analysis. These diagnostics can be complementary, but mixing them within a single comparison makes attribution difficult.
Assumptions, Limitations, and Common Mistakes
The comparison assumes that each run uses the same cell identities and that the clustering procedure and its settings are genuinely held fixed. It also assumes that the chosen perturbation scheme is relevant to the question. If the eligible gene pool is unsuitable, the sampled sets may test an uninteresting form of variability. If only one feature set is perturbed, there is little basis for describing the distribution of outcomes. Because correlated genes can carry overlapping information, resampling individual genes does not guarantee that independent biological information has been removed.
- Mistake: treating genes as independent replicates. Random gene resampling is a way to construct feature sets, not evidence that genes are independent observations. Consider correlated genes and, when justified, prespecified block perturbations.
- Mistake: changing several analysis choices at once. If both features and clustering resolution change, a change in the partition cannot be attributed specifically to features. Hold other choices fixed for this diagnostic.
- Mistake: calling a high ARI proof of true clusters. The ARI measures agreement between partitions under the tested perturbations. A consistently reproduced partition can still represent an artifact or an oversimplification of continuous cell structure.
- Mistake: reporting only an average. A favorable average can hide a consequential low-agreement run. Show the distribution or at least a robust summary together with the range and perturbation design.
The ARI is global, so similar overall scores can arise from different local changes. It also does not explain which features drove the difference, quantify assignment uncertainty, or account for uncertainty from sampling biological donors. Feature robustness is therefore one validity diagnostic among several, not a replacement for biological interpretation or other checks.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a feature-perturbation analysis, why should the same cells be used in every run?
- A perturbed partition has ARI 0.05 against the baseline. Does this mean 5% of cells kept their cluster assignments? Explain.
- Five ARIs are 0.84, 0.86, 0.87, 0.89, and 0.20. Find the median and state why reporting only the mean may be misleading.
- Why might omitting a correlated gene module answer a different question from randomly omitting the same number of individual genes?
Solutions. (1) Holding cells fixed isolates feature changes rather than mixing feature sensitivity with cell-composition changes. (2) No. ARI is an adjusted pair-based measure of agreement between partitions, not a percentage of cells retaining labels. (3) The median is 0.86; the low value of 0.20 signals a sensitive run that a mean alone could obscure. (4) Correlated genes can carry overlapping information, so omitting a whole module removes a coordinated set of features, while random individual omissions may leave much of that information represented.