Why Reporting Choices Matter
A clustering result is not fully described by a list of cluster labels or a single validity score. Its interpretation depends on choices made before and during clustering: which representation of the cells was used, what distance defined similarity, how the algorithm was tuned, and what was perturbed to assess stability. A compact report makes these choices visible so another analyst can understand what the reported evidence does and does not support.
The goal is not to find one number that certifies a partition. Internal validity metrics describe aspects of a partition under a specified geometry; stability summaries describe how assignments change under specified perturbations. Neither by itself establishes that the clusters are uniquely correct or biologically discrete. The introductory material on cluster validity and stability motivates these distinctions; here the emphasis is on reporting them clearly and reproducibly.
What a Compact Report Must Specify
Readers need enough information to reconstruct the analysis at the level of its statistical decisions. At minimum, report the following.
- Cells and representation: identify the cells included, preprocessing relevant to the analysis, and the representation used for clustering, such as selected features or a specified low-dimensional embedding. State the distance or similarity measure in that representation.
- Clustering procedure: name the algorithm, the selected number of clusters, important tuning parameters, and how initialization or random seeds were handled. If several choices were evaluated, describe the selection rule rather than reporting only the winning result.
- Validity metrics: name each metric, its direction of interpretation, its scale where relevant, and the representation or distance on which it was computed. Report the value and, when useful, its distribution across cells or clusters.
- Perturbation design: state what was resampled or changed, the sampling fraction or number of units, whether the pipeline was refit, how many runs were performed, and whether the same cells were compared across runs.
- Cluster matching and uncertainty: describe how labels were aligned across runs, what denominator defines each recovery proportion, and how between-run variation is summarized.
- Limitations: mention material instability, metric disagreement, geometric dependence, and the fact that reproducibility under a chosen perturbation is not proof that a partition is true.
A reader should not have to guess whether the analysis clustered cells in a particular embedding or in the original feature space. Likewise, a statement such as “clusters were stable under resampling” is incomplete unless it says what was resampled, how much data remained, how clusters were matched, and which cells entered the recovery calculation.
State the Metric and Its Geometry
For example, the silhouette of cell \(i\) compares its average dissimilarity \(a(i)\) to cells in its assigned cluster with the smallest average dissimilarity \(b(i)\) to another cluster. When both quantities are defined and not both zero, its silhouette is
The score lies between \(-1\) and \(1\): a larger value indicates that the cell is, under the chosen dissimilarity, closer on average to its assigned cluster than to its nearest alternative. A report should name the dissimilarity and representation. The score is not geometry-free: changing the embedding or distance can change both \(a(i)\) and \(b(i)\), and therefore the result.
For two partitions \(U\) and \(V\) of the same cells, variation of information is
where \(H\) is partition entropy and \(I\) is mutual information, computed using the same logarithm base throughout. With natural logarithms, VI is measured in nats; zero means the partitions agree exactly, and larger values indicate greater difference. VI is a comparison between two partitions, not a score with a universal threshold for acceptable stability. Include the comparison population and any normalization if one is used.
Worked Example: Reporting a Silhouette Calculation
Suppose a hypothetical cell has average within-cluster dissimilarity \(a(i)=0.42\) and smallest average dissimilarity to another cluster \(b(i)=0.70\), both measured using Euclidean distance in a stated two-dimensional embedding.
Substitute into the silhouette formula:
For this cell, the silhouette is 0.400 on the specified embedding and distance. The positive value says that its average dissimilarity to its assigned cluster is smaller than its average dissimilarity to the nearest alternative cluster. It does not establish that the cluster is biologically real. A useful report would also give the distribution of cell-level scores, since a moderate mean can conceal cells with negative scores or a small cluster with poor separation.
Design and Summarize Perturbation Analyses
Stability is defined relative to a perturbation design, not in the abstract. A perturbation might resample cells, resample donors, vary a random seed, or alter a pipeline choice. These designs answer different questions. If the aim is to assess sensitivity to sampling across donors, for example, the report must say whether donors—not merely cells—were resampled and whether sampling was stratified. State whether each run refit the clustering pipeline or only reassigned cells using an already fitted procedure.
For a reference cluster \(C_k\), a simple cluster-wise recovery proportion in a perturbed run is the number of its sampled cells assigned to its matched cluster, divided by the number of reference-cluster cells that were sampled and could therefore be assessed. If \(S\) is the set of cells shared with that run and \(C_k^*\) is the cluster matched to \(C_k\), this is
The denominator matters. It is not generally the full reference-cluster size when only a subset of cells is present in a run. The matching rule also matters: report whether clusters were aligned using maximum overlap or another prespecified rule, and how splits, merges, or unmatched clusters were handled. Matching labels is necessary because numerical cluster labels themselves have no intrinsic meaning.
Across perturbations, report the number of runs and a distributional summary, such as the median and interquartile range or selected quantiles, for each cluster. Report the range or a plot as appropriate when rare clusters or outlying runs are important. These summaries describe observed variability over the chosen perturbations; they are not automatically confidence intervals for an underlying population parameter. If a confidence interval is reported for a clearly defined estimand, its sampling model and assumptions need to be stated separately.
Worked Example: Cluster-Wise Recovery Under 80% Subsampling
In a hypothetical analysis, the reference partition contains 1,000 cells in three clusters: A has 500 cells, B has 400, and C has 100. Each perturbation is an 80% subsample of the 1,000 reference cells, so exactly 800 cells are included in a run. In one illustrative run, 400 A cells, 320 B cells and 80 C cells are included; these counts sum to 800.
After refitting the clustering and matching its clusters to the reference partition by maximum overlap, suppose 372 of the sampled A cells, 288 of the sampled B cells and 56 of the sampled C cells fall in their matched clusters. The recovery proportions are:
The run recovers 93.0% of sampled A cells and 90.0% of sampled B cells in their matched clusters, compared with 70.0% of sampled C cells. The denominators are the numbers of cells from each reference cluster present in this 80% subsample, not the full reference sizes. The result flags lower recovery for C in this run; it does not alone establish that C is invalid or that its recovery will be 70% in future data. A report should give the corresponding summaries across all perturbation runs and the number of cells available for each cluster in each run.
A Short Justification for the Recovery Denominator
Why divide by the number of sampled cells from the reference cluster rather than its full size? Let \(C_k\) be a reference cluster and \(S\) the cells present in a perturbed run. Only cells in \(C_k\cap S\) can be checked for recovery in that run. The matched cluster recovers the subset \(C_k\cap S\cap C_k^*\). Thus the fraction of assessable reference-cluster cells recovered is the size of that recovered subset divided by the size of the assessable set:
Dividing instead by \(|C_k|\), the full cluster size, would count deliberately omitted cells as failures. That would mix two quantities—sampling coverage and assignment recovery—and make the interpretation depend mechanically on the subsampling fraction.
From Results to an Auditable Summary
A compact table can place validity and stability results side by side, but it should not imply that the measures are interchangeable. Consider a hypothetical report with a mean silhouette of 0.31 and median VI of 0.12 nats between the reference partition and resampled runs. The silhouette summarizes separation under one chosen geometry, while VI summarizes partition disagreement across runs. Add cluster-wise recovery distributions because a global comparison can obscure a poorly recovered small cluster.
Worked Example: Interpreting Two Partitions with VI
Consider four hypothetical cells and two partitions. Partition \(U\) groups cells as \(\{1,2\}\) and \(\{3,4\}\). Partition \(V\) groups them as \(\{1,3\}\) and \(\{2,4\}\). Each partition has two equally sized groups, so its entropy using natural logarithms is \(\ln 2\). Every pair of one group from \(U\) and one group from \(V\) has one cell in common; consequently the joint group membership gives no information about which group a cell occupies in the other partition, and \(I(U;V)=0\).
Therefore:
This positive value reflects disagreement between these partitions. It should be reported with the fact that the comparison concerns the same four cells and uses natural logarithms. The calculation does not supply a universal cutoff for declaring a clustering stable; interpretation requires the perturbation context, run-to-run distribution and cluster-wise results.
Record the representation, distance, algorithm, cluster number, tuning choices and seed handling for the reference analysis.
State what is resampled or varied, the sampling fraction, the number of runs, and which parts of the pipeline are refit.
Describe the label-matching rule, the shared-cell set, recovery denominator, metric scale and any handling of splits or merges.
Show global and cluster-wise summaries across runs, note disagreements or weak clusters, and restrict conclusions to robustness under the tested choices.
Diagnostics, Alternatives and Limitations
Several warning signs deserve explicit attention. A high average validity score alongside a group of strongly negative cell-level silhouettes may indicate that the average hides poorly assigned cells. High overall agreement can coexist with weak recovery of a rare cluster. A cluster absent from some resamples has no recovery proportion for those runs; omission is a result to report, not a reason to silently exclude those runs. If donor-stratified and cell-level resampling give different results, describe the discrepancy rather than selecting the more favorable summary.
The main alternatives are complementary, not competing certificates. Internal validity metrics such as silhouette describe separation or compactness relative to a chosen distance geometry. Pairwise partition comparisons such as VI describe changes in labels across runs. Cluster-wise recovery localizes that change to particular reference clusters, while co-association summaries can describe how often cell pairs occur together without requiring one run’s labels to be treated as a universal reference. Report what each method measures and why it was selected. Metrics can disagree because they ask different questions or use different geometry; disagreement is informative and should remain visible.
Stability has important limits. Repeatedly obtaining the same result under a narrow set of perturbations can reflect a strong algorithmic preference or a shared preprocessing choice, not a uniquely correct biological partition. Conversely, instability can arise from a genuine continuum, low information, rare populations, or an unsuitable representation. External evidence and scientific interpretation are separate from internal validity and resampling stability. A stability report should therefore say “robust under these perturbations,” not “the true clusters have been discovered.”
Common Reporting Mistakes
- Wrong: “The clusters were stable because the overall agreement was high.” Correction: Name the comparison metric, perturbation design and matching rule; provide cluster-wise summaries so a weak or rare cluster is not hidden.
- Wrong: Treating cells omitted by subsampling as assignment errors. Correction: Calculate recovery among the reference-cluster cells actually present in the run, and report the sampling fraction separately.
- Wrong: Comparing silhouette values as though they were independent of the representation. Correction: State the embedding or feature space and distance used; geometry can change validity scores.
- Wrong: Calling a run-to-run range a confidence interval. Correction: Describe it as observed perturbation variability unless a statistical sampling interpretation and its assumptions have been justified.
- Wrong: Concluding that stable clusters are true clusters. Correction: Limit the claim to reproducibility under the tested pipeline and perturbations.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A reference cluster contains 120 cells. A run includes 90 of them, and 72 are assigned to its matched cluster. What is recovery, and why is 120 not the denominator?
- A cell has \(a(i)=0.30\) and \(b(i)=0.45\). Calculate its silhouette and state what a positive value means in this context.
- Why should a report give both an overall partition-comparison summary and cluster-wise recovery?
- A team reports low VI across repeated runs and concludes that the clusters are biologically true. What is the error in this conclusion?
Solutions. (1) Recovery is \(72/90=0.800\), or 80.0%. The 90 cells are the reference-cluster members assessable in that run; the 30 omitted cells were not tested for assignment recovery. (2) \(s(i)=(0.45-0.30)/0.45=0.333\). The positive value indicates that, under the specified dissimilarity, the cell is on average closer to its assigned cluster than to its nearest alternative. (3) An overall score can conceal poor recovery in a small or rare cluster; cluster-wise values reveal where disagreement occurs. (4) Low VI supports similar partitions under the tested runs and pipeline, not biological truth. The report must state the perturbations and acknowledge that stability is conditional on them.