Cluster-Wise Jaccard Stability

Why Measure Stability Cluster by Cluster?

A partition-level agreement score can summarize how similar two complete clusterings are, but it may not reveal which particular subgroup is recurring. This matters in multi-omics analyses: an overall score can conceal that one clinically or biologically important subgroup is reproducible while another is repeatedly rearranged. Cluster-wise Jaccard stability focuses on the overlap of one reference cluster with its counterpart in a resampled fit.

The comparison requires care. Resamples may contain different subjects, and cluster labels such as “cluster 1” are arbitrary across fits. First compare assignments on the subjects observed in both fits; then align clusters by their membership overlap. The result describes recurrence of observed memberships under the chosen resampling and clustering procedure. It does not establish that a cluster is biologically real, nor that it has a particular clinical meaning.

Definition: Let $C_1,\ldots,C_K$ be the reference clusters and $D_{b1},\ldots,D_{bL_b}$ the clusters from resample $b$. Let $E_b$ be the set of subject IDs with valid cluster assignments in both fits. For reference cluster $k$ and resampled cluster $\ell$, define $A_{bk}=C_k\cap E_b$ and $B_{b\ell}=D_{b\ell}\cap E_b$. Their Jaccard similarity is $$ J_{bk\ell}=\frac{|A_{bk}\cap B_{b\ell}|}{|A_{bk}\cup B_{b\ell}|}, $$ provided the union is nonempty. It is 1 for identical sets with a nonempty union, and 0 for disjoint sets with a nonempty union. If both restricted sets are empty, the formula is $0/0$ and the comparison is undefined unless a different convention is explicitly specified.

Constructing the Comparison

The evaluation set $E_b$ is essential. A subject omitted from a resample has no assignment there, so that subject cannot be treated as belonging to a different cluster. Restricting both fits to shared IDs ensures that the numerator and denominator compare like with like. The two fits should also use harmonized subject identifiers and the same unit of analysis. A row representing a patient in one fit must not be paired with a sample, visit, or aliquot in the other.

For ordinary Jaccard similarity, membership is set-based: a subject either belongs to a cluster or does not. If a bootstrap resample includes the same subject multiple times, count that subject once in the comparison unless a weighted Jaccard measure has been deliberately defined. That choice should be consistent across resamples and reported, because a multiplicity-weighted comparison is a different measure.

Cluster labels cannot be compared by their printed numbers. For a given resample, form the matrix of $J_{bk\ell}$ values and choose an alignment that maximizes the total similarity while matching each cluster at most once. When the numbers of clusters differ, an assignment can include dummy matches with score zero, making unmatched clusters explicit rather than forcing one cluster to represent multiple distinct reference clusters.

Matching rule: A one-to-one assignment $\pi$ pairs reference clusters with resampled clusters so that no cluster is used more than once. Choose an assignment maximizing $\sum_k J_{bk,\pi(k)}$, adding zero-score dummy clusters if needed when the cluster counts differ. This is appropriate when the goal is to track distinct cluster identities; it is not the same as independently choosing the best counterpart for every reference cluster.

Worked Example: Jaccard Similarity for One Cluster

Suppose the shared evaluation set contains subjects 1 through 8. A reference cluster contains subjects $A=\{1,2,3,4,5,6\}$, and a candidate resampled cluster contains $B=\{1,2,3,5,7,8\}$. Calculate their Jaccard similarity.

The intersection is $\{1,2,3,5\}$, so $|A\cap B|=4$. Each set has six subjects, and their union contains all eight evaluation subjects: $|A\cup B|=6+6-4=8$. Therefore, $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|} =\frac{4}{8}=0.500. $$

On this evaluation set, half of the subjects belonging to at least one of these two clusters belong to both. The score is an overlap summary, not the probability that the reference cluster is correct.

Why Matching Must Be One-to-One

If each reference cluster independently selected whichever resampled cluster gave its highest Jaccard score, two reference clusters could both claim the same counterpart. Such a result may be useful as a diagnostic of a split or merge, but it does not provide a consistent one-to-one alignment. Maximum-total matching instead makes the alignment decision jointly, allowing a cluster’s best local pairing to be traded against another pairing when necessary.

Worked Example: Choosing an Alignment

On a shared evaluation set of eight subjects, let the reference clusters be $A=\{1,2,3,4\}$ and $B=\{5,6,7,8\}$. Let resampled clusters be $U=\{1,2,3,5\}$ and $V=\{4,6,7,8\}$. Compute the Jaccard matrix and determine the best one-to-one match.

For $A$ and $U$, the intersection has three subjects and the union has $4+4-3=5$, giving $J(A,U)=3/5=0.600$. For $A$ and $V$, the intersection is $\{4\}$, so the union has $4+4-1=7$ subjects and $J(A,V)=1/7\approx0.1429$. By the same calculation, $J(B,U)=1/7\approx0.1429$, while $J(B,V)=3/5=0.600$.

	U	V
A	0.600	0.1429
B	0.1429	0.600

Matching $A$ to $U$ and $B$ to $V$ gives a total score of $0.600+0.600=1.200$. The alternative assignment gives $1/7+1/7=2/7\approx0.2857$. Thus the maximum-total one-to-one match is $A\leftrightarrow U$, $B\leftrightarrow V$. Each cluster has a matched Jaccard score of 0.600.

Summarizing Recurrence Across Resamples

After aligning each resampled fit to the reference, report the matched Jaccard score for each reference cluster across resamples. A simple summary is the arithmetic mean of that cluster’s defined scores, accompanied by the number of resamples contributing to the mean and a distributional summary such as the range or median. The mean describes average overlap under the particular resampling design, fitting procedure, and matching rule. It is not a universal property of the cluster independent of those choices.

A comparison is evaluable when its restricted sets have a nonempty union. If the reference cluster has no members in $E_b$ but its matched resampled cluster does, the intersection is empty and the union is nonempty, so the Jaccard score is 0. If the reference cluster has members but the matched resampled cluster is empty, the score is also 0. If both restricted sets are empty, the score is undefined under the formula; do not silently replace it with zero. State how many comparisons were evaluable and how undefined cases were handled. If the matching algorithm assigns a dummy counterpart, report that as unmatched according to the chosen convention rather than disguising it as an observed cluster.

Worked Example: Recurrence With an Unevaluable Comparison

A reference cluster is $C=\{a,b,c,d\}$. In three resamples, the shared evaluation set is $E_b=\{a,b,c,d\}$. After matching, the resampled counterparts are $D_1=\{a,b,c\}$, $D_2=\{a,b,d\}$, and $D_3=\varnothing$ on that evaluation set. Calculate the three scores and the mean of the defined scores.

For the first resample, the intersection has three subjects and the union is $C$, with four subjects. Thus $J_1=3/4=0.750$. For the second, the intersection is $\{a,b,d\}$, again with four subjects in the union, so $J_2=3/4=0.750$. For the third, the reference set has four members and the resampled set has none. The intersection is empty, but the union has four members, so $J_3=0/4=0$.

All three comparisons are evaluable; the mean is $(0.750+0.750+0)/3=0.500$. This low mean reflects a zero-overlap comparison in the third resample, not a missing comparison. If instead both restricted sets had been empty, that resample’s score would be undefined and would not enter the mean; the number of evaluable resamples would then need to be reported.

Assumptions, Diagnostics, and Alternatives

The arithmetic is simple, but interpretation depends on several conditions. Subject IDs must identify the same observational units in both fits; the reference and resampled partitions must be defined on comparable units; and the shared evaluation set must be constructed consistently. The resampling scheme and the rule for preprocessing, feature selection, and fitting determine what kind of perturbation the stability score reflects. Cluster-wise Jaccard measures the stability of assignments under that procedure, not stability under every possible analysis choice.

Check the evaluation-set size for each resample, the number of empty restricted clusters, and the number of undefined comparisons. Inspect the Jaccard matrix and the selected matches, especially when several entries are similar or tied. A maximum-total assignment can be non-unique under ties; report the tie-handling rule if it affects cluster-specific summaries. A cluster-count mismatch or repeated dummy match is also important context: it can indicate that a reference group lacks a distinct counterpart, rather than merely having a modest overlap score.

Cluster-wise Jaccard answers a different question from the Adjusted Rand Index for run-to-run agreement. The latter gives a partition-level summary, whereas Jaccard localizes overlap to matched clusters. Co-clustering matrices summarize pairwise co-membership across runs, while Jaccard summarizes the overlap of a selected cluster pair. These alternatives can complement one another: a single number may simplify comparisons, but can hide which subgroup is recurring. None establishes biological validity, and results can depend on the reference fit, resampling unit, cluster count, and evaluation-set composition.

Common pitfall: “The reference cluster is absent from the shared evaluation set, so its Jaccard score must be undefined.” That conclusion is not generally correct. If the matched resampled cluster has members in the evaluation set, the intersection is empty and the union is nonempty, so the score is 0. The score is undefined under the ordinary formula only when both restricted sets are empty. Report that distinction and the number of evaluable comparisons.

Practical Reporting

A useful report identifies the reference partition, the resampling design, the evaluation-set rule, the matching rule, and how unmatched or undefined comparisons were handled. For each reference cluster, give its matched Jaccard scores or a summary with the number of evaluable resamples. Retain enough detail to see whether a low summary arises from consistently modest overlap, occasional zero-overlap runs, or a small number of evaluable comparisons. Avoid describing a high score as a probability that the subgroup is “true”; it is evidence only of repeated membership overlap under the stated analysis.

Check Your Understanding

Answer each question, then compare with the solutions below.

A reference cluster has 9 members and its matched resampled cluster has 7 members; 5 subjects are in both. What is the Jaccard score?
Why can independently choosing the best counterpart for every reference cluster fail to produce a one-to-one alignment?
The reference cluster is empty after restriction to the shared evaluation set, but its matched resampled cluster has members there. Is the score undefined or zero? What if both restricted sets are empty?
A matched Jaccard score is 0.60. State its interpretation without treating it as a probability that the cluster is correct.

Solutions. (1) The union has $9+7-5=11$ subjects, so $J=5/11\approx0.4545$. (2) Two reference clusters may select the same resampled cluster; a maximum-total one-to-one assignment prevents a resampled cluster from being used more than once when distinct cluster identities are being aligned. (3) If the resampled cluster is nonempty, the intersection is empty but the union is nonempty, so the score is 0. If both restricted sets are empty, the formula is undefined unless a different convention is explicitly specified; report it as unevaluable and state the evaluable count. (4) On the stated evaluation set, 60% of subjects in the union of the reference cluster and its matched resampled counterpart belong to both sets. This describes overlap, not the probability that the cluster is true.

What You'll Learn