Comparing Clusters With Reference Labels

Why Compare Clusters With Reference Labels?

A clustering partitions cells using expression patterns, while a reference label assigns names or categories using external annotation. Comparing the two can indicate whether the discovered groups align with an existing cell-type scheme. It does not establish that the reference is complete or biologically definitive: the score describes agreement between two assignments on the cells being compared.

Two common measures are the adjusted Rand index (ARI), which compares pairs of cells, and normalized mutual information (NMI), which compares the information shared by the two partitions. Both ignore the literal names of labels. A cluster called “1” can match a reference category called “T cell” just as well as a cluster called “T cell” if the cell memberships agree.

Contingency Table and Definitions

Suppose $n$ cells each have a cluster assignment and a reference label. Let $n_{ab}$ be the number assigned to cluster $a$ and reference category $b$. The row and column totals are $n_{a\cdot}=\sum_b n_{ab}$ and $n_{\cdot b}=\sum_a n_{ab}$. This table is the starting point for both scores.

Definition: The adjusted Rand index compares the number of cell pairs placed together in both partitions with the number expected under a fixed-margins chance-agreement reference. Writing $\binom{m}{2}=m(m-1)/2$, its usual form is $$ \operatorname{ARI}= \frac{ \sum_{a,b}\binom{n_{ab}}{2} -\dfrac{\left(\sum_a\binom{n_{a\cdot}}{2}\right) \left(\sum_b\binom{n_{\cdot b}}{2}\right)}{\binom{n}{2}} }{ \dfrac{1}{2}\left[ \sum_a\binom{n_{a\cdot}}{2}+ \sum_b\binom{n_{\cdot b}}{2} \right] -\dfrac{\left(\sum_a\binom{n_{a\cdot}}{2}\right) \left(\sum_b\binom{n_{\cdot b}}{2}\right)}{\binom{n}{2}} }. $$ The index is 1 for identical partitions, can be near 0 or negative for weak agreement relative to its adjustment, and is not a p-value.

Definition: The normalized mutual information compares the mutual information $I(C;R)$ between cluster assignment $C$ and reference label $R$ with the entropies of those assignments. Using natural logarithms and the geometric-mean normalization, $$ I(C;R)=\sum_{a,b:\,n_{ab}>0}\frac{n_{ab}}{n} \log\left(\frac{n\,n_{ab}}{n_{a\cdot}n_{\cdot b}}\right), \qquad H(C)=-\sum_a\frac{n_{a\cdot}}{n}\log\left(\frac{n_{a\cdot}}{n}\right), \qquad H(R)=-\sum_b\frac{n_{\cdot b}}{n}\log\left(\frac{n_{\cdot b}}{n}\right), $$ $$ \operatorname{NMI}(C,R)=\frac{I(C;R)}{\sqrt{H(C)H(R)}}. $$ For non-degenerate partitions, NMI ranges from 0 to 1. Terms with zero counts contribute zero. If both partitions consist of one group, define NMI as 1; if exactly one does, define it as 0.

ARI and NMI are defined for hard, discrete assignments of the same cells. They treat labels as categories, not as ordered measurements. ARI requires at least two cells, and its displayed denominator must be nonzero; in degenerate cases, implementations may use explicit conventions, which should be reported. For NMI, the edge-case conventions above make constant-partition comparisons interpretable.

How the Scores Behave

A brief identity explains one useful property of NMI. Mutual information can be written as $I(C;R)=H(R)-H(R\mid C)$, where $H(R\mid C)$ is the remaining uncertainty about the reference label after the cluster is known. If every cluster belongs to only one reference category, then $H(R\mid C)=0$, so $I(C;R)=H(R)$. Thus a clustering that subdivides reference categories without mixing them can retain all information about the reference, even though it has more groups. That distinction matters when interpreting a high NMI.

For ARI, the pair-count terms have a direct reading: $\sum_{a,b}\binom{n_{ab}}{2}$ counts cell pairs grouped together in both assignments. The adjustment uses the cluster and reference group sizes. The details of why that adjustment is made are a separate question; here, use ARI as an agreement summary rather than as evidence for a hypothesis test.

Worked Example: Exact Agreement Despite Different Label Names

Suppose four hypothetical cells have reference labels A, A, B, B, while the clustering labels them 7, 7, 3, 3. The contingency table has two nonzero cells, each with count 2; both row and column totals are 2.

For ARI, the common-pair count is $2\binom{2}{2}=2$, and each partition has $\sum\binom{2}{2}=2$. Since $\binom{4}{2}=6$, the adjusted expected term is $2(2)/6=0.6667$. Therefore,

\operatorname{ARI}=\frac{2-0.6667}{\frac12(2+2)-0.6667} =\frac{1.3333}{1.3333}=1.

Each partition has entropy $-2(0.5\log 0.5)=0.6931$, and the label determines the other partition exactly, so $I(C;R)=0.6931$. Hence $\operatorname{NMI}=0.6931/\sqrt{0.6931(0.6931)}=1$. The numeric label names do not affect either result.

Worked Example: A Partition That Mixes the Reference Categories

Consider eight hypothetical cells: four have reference A and four have reference B. Two clusters each contain two A cells and two B cells. The contingency table is therefore $\begin{bmatrix}2&2\\2&2\end{bmatrix}$.

The common-pair count is $4\binom{2}{2}=4$. The cluster-pair total and reference-pair total are each $2\binom{4}{2}=12$, and $\binom{8}{2}=28$. Thus the adjusted term is $12(12)/28=5.1429$, and

\operatorname{ARI}= \frac{4-5.1429}{\frac12(12+12)-5.1429} =\frac{-1.1429}{6.8571}=-0.1667.

Each cluster has the same 50:50 reference composition, so knowing the cluster does not change uncertainty about the reference: $I(C;R)=0$. Both entropies are $\log 2=0.6931$, giving $\operatorname{NMI}=0/0.6931=0$. These values indicate that the assignments do not align in this example; they do not indicate a probability that the reference labels are wrong.

Granularity and Annotation Coverage

Reference labels and clusters need not operate at the same level of detail. A broad reference label such as “myeloid” may encompass several finer clusters. Conversely, a coarse clustering may merge reference categories that an annotation distinguishes. The contingency table exposes these patterns: rows split across columns indicate mixed clusters; columns spread across rows indicate reference categories subdivided by the clustering.

A score compresses this structure into one number. In particular, NMI can be high when clusters cleanly subdivide reference categories, because the fine cluster assignment may still identify the broader reference label. That is not the same as one-to-one cluster matching. Always inspect group sizes and the contingency table, and state whether the intended comparison concerns broad identity, fine subtypes, or one-to-one correspondence.

Worked Example: Fine Clusters Within a Broad Reference Type

Suppose eight hypothetical cells have four reference-A cells and four reference-B cells. The clustering creates three groups: two contain two A cells each, and the third contains all four B cells. The contingency table has counts $2,2,4$ in those three matching cells.

The common-pair count is $1+1+6=8$. Cluster pair totals sum to 8, reference pair totals sum to 12, and $\binom{8}{2}=28$. The adjustment is $8(12)/28=3.4286$, so

\operatorname{ARI}=\frac{8-3.4286}{\frac12(8+12)-3.4286} =\frac{4.5714}{6.5714}=0.6957.

The cluster entropy is $-2(0.25\log0.25)-0.5\log0.5=1.0397$, while reference entropy is $0.6931$. Every cluster contains only one reference type, so $I(C;R)=H(R)=0.6931$, and

\operatorname{NMI}=\frac{0.6931}{\sqrt{1.0397(0.6931)}}=0.8165.

The scores show substantial agreement, but not identical partitions: reference A has been split into two clusters. If that split is biologically meaningful, the broad reference may be too coarse to assess subtype discovery; if one-to-one agreement is the goal, the split is a discrepancy.

Incomplete annotation requires a separate decision. A missing label is not automatically a biological category called “unknown.” If the goal is agreement with known annotations, compute the scores on cells with observed reference labels and report how many cells were included and excluded. This estimates agreement on the annotated subset, which may not represent all cells if annotation availability is selective.

Worked Example: Restricting Comparison to Annotated Cells

Suppose six hypothetical cells are clustered as X, X, Y, Y, Z, Z. The first four have reference labels A, A, B, B; the last two are unannotated. On the four annotated cells, clusters X and Y each contain only their corresponding reference type, so the two restricted partitions match exactly.

For those four cells, the ARI calculation is $[2-2(2)/6]/[\frac12(2+2)-2(2)/6]=1$. Each partition has two groups of size 2, so $I=H(C)=H(R)=0.6931$ and NMI is 1. The correct conclusion is perfect agreement among the four annotated cells, not perfect agreement across all six cells. The two missing labels provide no evidence about their reference types.

Diagnostics, Alternatives, and Limitations

Before interpreting either index, check that the cell sets are identical and that each included cell has exactly one assignment in each partition. Then examine the contingency table for rare categories, large mixed groups, unexpected splits, and missingness. A single score can be dominated by large groups, while small but important populations may contribute little to an overall summary. Report group counts alongside the metric.

Do not treat a score as a universal pass/fail threshold. Its meaning depends on the number and sizes of groups, annotation granularity, and the intended use of the reference. Neither ARI nor NMI quantifies annotation certainty, tests a biological hypothesis, or proves that clusters correspond to causal or mechanistic cell types.

Other summaries answer different questions. Purity reports the fraction of cells in each cluster belonging to its most common reference category, but can look favorable when many tiny clusters are created. Classification accuracy requires matching cluster names to reference labels and is awkward when the number of groups differs. An adjusted or normalized information score provides label-invariant comparison, but still compresses the table. Use the contingency table and the relevant alternative when the pattern matters more than a single aggregate.

A common mistake is to treat every unannotated cell as one shared “unknown” reference class and calculate agreement over all cells. This makes missingness behave like a real category: two unrelated cells may be counted as agreeing simply because both lack labels. Instead, define the evaluation set from observed reference labels, report its size and selection caveat, and analyze unannotated cells separately if needed.

Key takeaway: ARI and NMI summarize agreement between two partitions, not truth. Interpret each with the contingency table, the chosen label granularity, and an explicit account of which cells have usable reference annotations.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

Why can two partitions with different label names have ARI equal to 1?
What does an NMI of 0 mean for two non-constant partitions?
A clustering separates one broad reference category into several pure clusters. Why might NMI remain high?
What should be reported when reference labels are available for only a subset of cells?

Solutions. 1. The indices use cell memberships and ignore the literal names. 2. The assignments share no mutual information in the observed contingency table. 3. The fine cluster can still determine the broad reference category, so little or no reference uncertainty remains given the cluster. 4. Report the annotated-cell count or fraction, compute agreement on those cells, and note that selective annotation may limit generalization to unannotated cells.