Why Perturb Features?
A discovered subtype can appear stable when the individuals or train-test splits are perturbed, yet depend heavily on a small set of measured features. For example, a cluster may be separated by a handful of transcripts while metabolite and protein measurements contribute little to its assignments. Perturbing features asks a different question from resampling individuals: if the measured feature set changes, do the same individuals remain grouped together?
The earlier material on repeated train-test perturbations for clusters considers assignment changes under sample splits. Here, keep the individuals fixed and change which features enter the analysis. This pairing helps isolate sensitivity to feature choice, provided the clustering procedure and other analysis choices are held constant. It does not establish that a subtype is biologically real, nor does it measure uncertainty from sampling a new population.
Designing a Feature Perturbation
A perturbation is informative only in relation to a clearly stated question. Randomly retaining, say, 80% of features probes robustness to broadly distributed feature loss. Removing an entire modality, such as the transcriptomic block, probes dependence on that modality. Removing a prespecified pathway or other feature block asks whether the result depends on that block. These designs answer different questions: random deletion can leave a large block mostly intact, whereas targeted deletion directly tests that block’s contribution.
For a comparison focused on features, use the same individuals in the reference and perturbed analyses. Keep the clustering method, number of groups, distance definition, and other settings fixed unless the stated perturbation specifically concerns one of them. A change in several choices at once makes it difficult to attribute changed assignments to feature removal. If the clustering algorithm is stochastic, repeat runs or control its random seed so that algorithmic randomness can be distinguished from feature sensitivity.
The feature perturbation scheme should also be specified before interpreting the results: name the feature blocks or sampling rule, state how many perturbations will be run, and record the number of features retained each time. When features are strongly correlated, deleting one feature may have little effect because a correlated feature carries similar information. Conversely, removing an entire correlated block may have a substantial effect. Thus, feature-level and block-level perturbations are not interchangeable.
This comparison is conditional on how the data were prepared. If the same preprocessing is held fixed, the result describes sensitivity to feature choice given that preprocessing. Re-estimating preprocessing within each perturbation changes the question and is treated separately in the later material on refitting preprocessing within resamples. State which choice was made rather than describing either result as unconditional robustness.
Comparing Assignments by Overlap
Let the reference partition have groups \(C_1,\ldots,C_K\), and let one perturbed partition have groups \(D_1,\ldots,D_L\), all assigned on the same individuals. Define \(n_k=|C_k|\), \(m_l=|D_l|\), and \(n_{kl}=|C_k \cap D_l|\). The overlap table with entries \(n_{kl}\) records how many individuals from each reference group occur in each perturbed group.
Retention asks how much of a reference group stays together; purity asks how specifically a perturbed group represents a reference group. A high retention can occur when the perturbed group also contains many individuals from other reference groups, giving low purity. These quantities do not require cluster labels to have the same names across runs: they depend on set intersections, not on whether an algorithm called a group “1” or “2.”
The maximum in the retention definition can be attained by more than one perturbed group. If so, report the tie. The retention value remains well-defined, but there is no unique best-overlapping group to call its match. Reporting the purity of only one tied group can give a misleading impression of a unique correspondence.
A Short Justification for the Overlap Summary
Why use the largest overlap divided by the reference group’s size? For a fixed \(C_k\), the quantities \(n_{kl}/n_k\) give the fraction of its members in each perturbed group. The largest such fraction is the largest portion that remains together in any single group. Because each individual belongs to exactly one \(D_l\), the fractions sum to one:
It follows that \(1/L \leq R_k \leq 1\) when there are \(L\) nonempty perturbed groups: at least one of \(L\) nonnegative fractions summing to one is at least \(1/L\), and none can exceed one. This bound is descriptive, not a universal pass/fail threshold; the number and sizes of perturbed groups affect what values are plausible. Dividing by \(m_l\) instead gives \(Q_{kl}\), answering the different question of how much of \(D_l\) comes from \(C_k\).
Worked Example: A Tied Best Overlap Must Be Reported
Suppose eight hypothetical individuals have reference groups \(A=\{A1,A2,A3,A4\}\) and \(B=\{B1,B2,B3,B4\}\). After a transcript-feature block is removed, the perturbed groups are \(A'=\{A1,A2\}\) and \(B'=\{A3,A4,B1,B2,B3,B4\}\). Calculate retention for reference group \(A\), and assess whether it has a unique best-overlapping perturbed group.
The overlaps are \(|A\cap A'|=2\) and \(|A\cap B'|=2\). Therefore,
The maximum is tied: neither \(A'\) nor \(B'\) is a unique best-overlapping group for \(A\). If the compositions of both tied groups are reported, their purities with respect to \(A\) differ:
Thus, two of the four reference \(A\) individuals remain together in \(A'\), while the other two are in \(B'\). The 100% purity of \(A'\) does not make \(A'\) the unique match; it is one of two tied groups by overlap, and the other tied group contains four \(B\) individuals as well.
Summarizing Repeated Feature Perturbations
For \(B\) perturbation runs, calculate \(R_k^{(b)}\) separately for each run \(b\), using that run’s perturbed groups. The mean retention for reference group \(C_k\) is
This is an average over the specified perturbations, not a probability that the group will be recovered in a new study. Report the spread or the individual run values as well as the mean: an average can conceal occasional severe fragmentation. If perturbation runs use different schemes, summarize each scheme separately rather than combining them into one average with no clear interpretation.
Worked Example: Comparing Retention Across Four Perturbations
In a hypothetical analysis, reference group \(A\) has six individuals and reference group \(B\) has six. Four feature perturbations produce the following largest overlap counts for each reference group. In each run, divide the count by the size of the relevant reference group to obtain retention.
| Run | Largest overlap for A | \(R_A^{(b)}\) | Largest overlap for B | \(R_B^{(b)}\) |
|---|---|---|---|---|
| 1 | 5 of 6 | 5/6 | 6 of 6 | 6/6 |
| 2 | 5 of 6 | 5/6 | 6 of 6 | 6/6 |
| 3 | 4 of 6 | 4/6 | 6 of 6 | 6/6 |
| 4 | 2 of 6 | 2/6 | 5 of 6 | 5/6 |
For group \(A\), the mean is
For group \(B\), the mean is
Across these four perturbations, \(B\) has higher mean retention than \(A\). However, its retention is not perfect in every run, and \(A\)’s mean does not show whether the low value occurred under a particular feature deletion. The run-specific results and the exact perturbation scheme are needed to interpret this contrast.
Testing Dependence on a Feature Block
To investigate whether a subtype depends on a narrow feature set, compare a broad perturbation scheme with a targeted one. Randomly sampled feature subsets provide a reference for the effects of ordinary feature loss; removing a modality or prespecified block tests dependence on that particular block. The same individuals should be used in both schemes if the aim is to separate feature sensitivity from sample sensitivity.
Worked Example: Random Subsets Versus Targeted Modality Removal
Consider a hypothetical reference group \(A\) of five individuals and a reference group \(B\) of five. In ten random-subset runs, the largest overlap for \(A\) is four individuals in eight runs and three individuals in two runs. In ten runs that remove the transcriptomic block, the largest overlap for \(A\) is one individual in six runs and two individuals in four runs. Suppose the corresponding largest overlaps for \(B\) are five in seven random-subset runs and four in three; under transcriptomic-block removal they are four in eight runs and three in two. Each count is divided by five because each reference group has five members.
For group \(A\), mean retention under random subsets is
Under transcriptomic-block removal, its mean retention is
For comparison, the same calculations for group \(B\) give
In this hypothetical set of runs, both groups have lower mean retention when the transcriptomic block is removed, especially group \(A\). This pattern suggests that the assignments of \(A\) are more sensitive to that targeted perturbation than to the stated random subsets. It does not prove that transcripts cause the subtype, or that the block alone explains it: correlated features, preprocessing, and the chosen clustering method can all affect the result.
Diagnostics, Alternatives, and Limits
Inspect the overlap table and group sizes, not only the average retention. A group that splits across several perturbed groups may have low retention even if no single overlap looks exceptional. A large perturbed group can have high overlap with a reference group but low purity because it also contains many other individuals. Ties in maximum overlap deserve explicit attention, and small reference groups require care: moving just one individual can change retention substantially.
- Check the perturbation actually occurred. Record the number of features retained and verify that a targeted block was fully excluded.
- Separate sources of change. Keep the individuals and analysis settings fixed when isolating feature sensitivity; if algorithmic randomness is relevant, assess it separately.
- Inspect each run. A mean can hide a run with a dramatic loss of retention, and different blocks can have different effects.
- Check for indirect feature dependence. Strongly correlated features can substitute for removed features, so a mild response to single-feature deletion does not rule out dependence on a broader block.
- Report ties and changing group sizes. They can make a single “matched group” interpretation ambiguous even when a numerical retention is easy to calculate.
Alternative perturbations answer complementary questions. Individual subsampling changes which samples are available and therefore does not isolate feature sensitivity. Random feature subsampling assesses robustness to the chosen deletion distribution but may leave a suspected block largely represented. Targeted block removal is more direct for a specified modality, but conclusions apply to that block and analysis setup. A full-pipeline perturbation can assess a broader workflow, but it combines effects from multiple changing steps. No one summary replaces examination of the perturbation design and assignment overlaps.
Feature-perturbation stability has important limits. It is conditional on the cohort, the feature set, the preprocessing choices, the clustering method, and the perturbations actually tried. High retention is not evidence that the groups correspond to disease mechanisms; low retention does not by itself show which feature is responsible. Stability is evidence about sensitivity of the procedure, not validation of the scientific interpretation.
Common Mistakes
- Treating a tied overlap as a unique match. The maximum overlap can be attained by several perturbed groups. Report the tie and, if relevant, each tied group’s purity.
- Confusing retention with purity. Retention divides by the reference-group size; purity divides by the perturbed-group size. They answer different questions.
- Calling a perturbation average a probability. A mean over chosen feature subsets summarizes those perturbations; it is not the probability of recovery in a new population.
- Attributing all assignment changes to features. Changes in individuals, preprocessing, or algorithm settings can also change assignments. Hold them fixed when the goal is feature-specific sensitivity.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A reference group has six individuals. In a perturbed run, its largest overlaps with the perturbed groups are four and two. What is its retention?
- A reference group of four has overlap two with each of two perturbed groups. Is there a unique best-overlapping group? What should be reported?
- Why might removing one feature at a time fail to reveal dependence on a correlated feature block?
- What does a mean retention of 0.80 across a prespecified set of feature perturbations establish, and what does it not establish?
Solutions. (1) \(R=\max(4/6,2/6)=4/6\approx0.667\). (2) No. The maximum overlap is tied at \(2/4=0.50\); report both tied groups and do not designate either as the unique match. If purity is relevant, calculate it separately for each tied group using that group’s size. (3) A correlated feature can partly replace the removed feature, preserving assignments even when the broader block carries important information; targeted block removal tests a different perturbation. (4) It summarizes the average largest fraction of each reference group retained together across those perturbations. It is not a probability of recovery in a new population, proof of biological validity, or evidence about perturbations that were not tested.