Metrics for Biological Conservation

What Biological-Preservation Metrics Measure

An integration can make cells from different batches more comparable while also changing which cell identities or biological relationships remain distinguishable. Biological-preservation metrics assess the latter question: does the integrated representation retain biological information that a defensible reference says should be present? They do not establish that the representation is biologically correct in every respect. Their conclusions depend on what the reference represents and on which aspect of biology the metric measures.

Three useful families are label agreement, neighborhood conservation, and feature-based criteria. Label agreement asks whether cells retain known identities. Neighborhood metrics ask whether biologically related cells remain near one another. Feature-based metrics ask whether selected molecular signals or contrasts remain detectable. These are different targets, so their numerical values are not directly interchangeable.

Definition: A biological-preservation metric is a numerical summary comparing an integrated result with a specified biological reference or criterion. The reference may be trusted cell labels, known cell-to-cell relationships, or predefined molecular features and contrasts. A metric is interpretable only with its reference, scale, and direction stated.

Before calculating any of these metrics, specify the cells being evaluated, the biological reference, and the representation being scored. A reference label should be sufficiently reliable for the question, and cell identities must be matched correctly between the reference and integrated result. If the reference is itself uncertain, agreement with it measures reproduction of that uncertainty as well as biology. Metrics computed over cells also do not make cells independent experimental replicates; a high score is descriptive evidence, not by itself a population-level inferential result.

Label Agreement

If each cell has a trusted reference label and the integrated analysis assigns a label using a specified rule, agreement can be summarized by accuracy: the fraction of cells whose assigned label equals the reference label. This requires the label names to mean the same thing in both sources. Accuracy is easy to interpret, but it can be dominated by common cell types and conceal poor performance for rare types.

\text{Accuracy}=\frac{\text{number of matching labels}}{n} =\frac{\sum_i n_{ii}}{n}

Here $n_{ij}$ is the number of cells in reference category $i$ assigned to category $j$, and the categories have already been aligned by meaning. When comparing clusterings, names such as “cluster 1” and “cluster 2” are arbitrary: exchanging those names must not change the agreement score. The adjusted Rand index (ARI) addresses this by comparing pairs of cells rather than requiring matching names.

Definition: For a contingency table with row totals $a_i$, column totals $b_j$, and $n_{ij}$ in cell $i,j$, let $M=\binom{n}{2}$, $A=\sum_i\binom{a_i}{2}$, $B=\sum_j\binom{b_j}{2}$, and $C=\sum_{i,j}\binom{n_{ij}}{2}$. When the denominator is nonzero, the adjusted Rand index is $$ \mathrm{ARI}= \frac{C-AB/M}{\tfrac{1}{2}(A+B)-AB/M}. $$ It is 1 for identical partitions, has expected value 0 under the usual random-partition model with fixed group sizes, and can be negative when agreement is below that expectation.

The adjustment follows from a pair-counting argument. There are $M$ possible cell pairs. If one partition groups $A$ pairs together and the other groups $B$ pairs together, then under random pairing with those group sizes fixed, the expected number together in both is $AB/M$: each of the $A$ pairs has probability $B/M$ of belonging to the other partition’s together-pairs. Subtracting this expectation centers the shared-pair count at zero under that reference model. The denominator scales the adjusted count by the average of the two partitions’ maximum together-pair counts, also after chance adjustment. This explains both the chance correction and why ARI is not simply a percentage of correctly labelled cells.

Worked Example: Accuracy and ARI From a Label Table

Suppose 20 hypothetical cells have three trusted reference labels and three assigned labels. The entries below give counts, with rows as reference labels and columns as assigned labels.

	Assigned A	Assigned B	Assigned C	Row total
Reference A	7	1	0	8
Reference B	1	6	0	7
Reference C	0	1	4	5

Accuracy. There are $7+6+4=17$ matches, so accuracy is $17/20=0.850$, or 85.0%. In this hypothetical sample, 85.0% of cells received their reference label.

ARI. The row totals are 8, 7, and 5; the column totals are 8, 8, and 4. Thus $A=\binom{8}{2}+\binom{7}{2}+\binom{5}{2}=59$, $B=\binom{8}{2}+\binom{8}{2}+\binom{4}{2}=62$, and $C=\binom{7}{2}+\binom{1}{2}+\binom{1}{2}+\binom{6}{2}+\binom{1}{2}+\binom{4}{2}=42$. Also $M=\binom{20}{2}=190$. Substitution gives

\mathrm{ARI}= \frac{42-(59)(62)/190} {\tfrac{1}{2}(59+62)-(59)(62)/190} =\frac{22.7474}{41.2474} \approx 0.5515.

The ARI is positive but below 1, indicating partial pairwise partition agreement. It is not an estimate that 55.15% of cells are correct; accuracy gives the cell-level matching fraction here.

Neighborhood Conservation

A label does not describe every relationship among cells. A neighborhood criterion instead asks whether cells that are biologically close according to a reference remain neighbors in the integrated representation. To make this comparison meaningful, define the reference neighborhoods independently of the integrated coordinates—for example, from known simulated truth, a trusted modality, or a prespecified biological reference. Simply treating the unintegrated graph as biological truth can be misleading when that graph also contains batch structure.

Let $N_{\mathrm{ref}}(i)$ and $N_{\mathrm{int}}(i)$ be the sets of $k$ reference and integrated nearest neighbors of cell $i$, respectively. A simple per-cell overlap is their Jaccard similarity, the number of shared neighbors divided by the size of their union. The average over cells gives a summary on a zero-to-one scale.

J_i= \frac{|N_{\mathrm{ref}}(i)\cap N_{\mathrm{int}}(i)|} {|N_{\mathrm{ref}}(i)\cup N_{\mathrm{int}}(i)|}, \qquad \overline{J}=\frac{1}{n}\sum_{i=1}^{n}J_i.

If both sets contain $k$ neighbors and share $r$, their union contains $k+k-r=2k-r$. Therefore $J_i=r/(2k-r)$, not $r/k$. This distinction matters because the Jaccard denominator counts every distinct neighbor appearing in either set. The score depends on $k$, the distance metric, tie handling, and the reference graph; those choices should be stated when reporting it.

Worked Example: Mean Jaccard Neighborhood Overlap

For three hypothetical cells, use $k=4$ and the following reference and integrated neighbor sets. Assume ties have been resolved by the same prespecified rule in both representations.

For cell 1, the sets are $\{2,3,4,5\}$ and $\{2,3,6,7\}$. They share 2 neighbors and have a union of 6, so $J_1=2/6=0.3333$. For cell 2, the sets are $\{1,3,4,5\}$ and $\{1,3,4,6\}$; the intersection has 3 members and the union has 5, so $J_2=3/5=0.6000$. For cell 3, the sets are identical, so $J_3=4/4=1.000$.

The mean is $\overline{J}=(0.3333+0.6000+1.000)/3=0.6444$. Thus, averaged across these three cells, the reference and integrated neighborhoods have Jaccard overlap 0.6444. The result describes this specific graph comparison; it does not prove that the reference graph captures every relevant biological relationship.

Feature-Based Criteria

Feature-based criteria evaluate selected measurements, such as predefined marker genes or known biological contrasts. A basic approach compares a feature summary in the reference with the corresponding summary after integration. For example, one may assess whether a set of prespecified cell-type contrasts has the same direction, or compare the vectors of reference and integrated contrast estimates using correlation.

The measurement scale is essential. A corrected embedding is a coordinate system, not a gene-expression matrix; its coordinates generally cannot be interpreted as expression values or log fold changes. A feature-based criterion should use an assay or summary for which the feature has a valid meaning, and should state whether the compared values are on the same scale. If the integrated output changes scale, comparing signs or rankings may be meaningful where comparing magnitudes is not.

Worked Example: Marker-Contrast Direction and Magnitude

Suppose three hypothetical, prespecified marker contrasts have reference log2 fold changes $x=(2,1,-1)$ and integrated-assay log2 fold changes $y=(1,0.5,-0.5)$. Assume these values are calculated using the same cell groups and a scale on which log2 fold changes are interpretable.

All three contrasts retain their direction: $2$ and $1$ remain positive, while $-1$ remains negative. Directional concordance is therefore $3/3=1.000$, or 100%. The integrated contrast magnitudes are each half their reference values: $1/2=0.5$, $0.5/1=0.5$, and $(-0.5)/(-1)=0.5$. A least-squares slope through the origin is

\hat{\beta}=\frac{\sum_j x_jy_j}{\sum_j x_j^2} =\frac{(2)(1)+(1)(0.5)+(-1)(-0.5)}{2^2+1^2+(-1)^2} =\frac{3}{6}=0.5000.

Here the integrated contrasts retain direction but are attenuated to half the reference magnitude. Their Pearson correlation is 1 because $y=0.5x$ exactly; correlation alone would therefore miss the magnitude reduction. Reporting direction, correlation, and a magnitude summary answers different questions.

Diagnostics, Comparisons, and Limitations

The three metric families are complementary rather than substitutes. Label agreement is direct when labels are trustworthy, but it compresses within-label structure and can hide errors in rare categories. Neighborhood overlap can capture local organization without requiring every cell to have a discrete identity, but it is sensitive to graph construction and the chosen reference. Feature criteria link preservation to interpretable molecular signals, but selected markers are an incomplete view of biology and their estimates can be noisy. A high value in one family does not entail a high value in another.

Metric family	Question answered	Important limitation
Label agreement	Do assigned identities match trusted cell labels?	Depends on reference quality; overall accuracy can be dominated by common labels.
Neighborhood overlap	Are reference neighbors retained around each cell?	Depends on the reference graph, distance, and neighborhood size.
Feature-based criteria	Are specified markers or contrasts retained?	Depends on feature selection and a valid, comparable measurement scale.

Several checks help reveal when a score is misleading. For label metrics, inspect the contingency table and per-label counts, not only overall accuracy; confirm that label meanings are aligned. For ARI, check whether the partitions contain enough structure for the denominator to be defined. In degenerate cases, such as partitions with no informative pair distinctions, the denominator can be zero, and software conventions may differ; report the case rather than silently treating it as an ordinary score. For neighborhood metrics, vary $k$ as a sensitivity check and inspect whether ties or highly uneven cell density drive the result. For feature metrics, prespecify the features where possible, verify the scale, and examine whether a small number of genes dominate the summary.

A common mistake is to report a high label accuracy as proof that all biology was preserved. That reasoning is too strong: accuracy evaluates agreement with the supplied labels, which may omit continuous state variation, rare subtypes, or molecular contrasts within a label. The correction is to state the target precisely, inspect class-specific results, and use additional metrics only when they answer a distinct, justified biological question. Likewise, these descriptive scores should not be treated as p-values or as evidence of causal effects.

Key takeaway: Interpret each preservation metric relative to its biological reference and exact target. Report the metric definition, important construction choices, and relevant subgroup or feature results; do not interpret one score as a complete measure of biological fidelity.

Check Your Understanding

Work through the questions, then compare your reasoning with the solutions.

A label table has 50 cells, of which 42 have matching reference and assigned labels. What is the accuracy, and what does it describe?
Two neighborhood sets each contain 5 cells and share 3. What is their Jaccard similarity?
Why can a Pearson correlation of 1 between reference and integrated feature contrasts fail to show that contrast magnitudes were preserved?
Name one reason that a high overall label accuracy could conceal a biological-preservation problem.

Solutions. (1) Accuracy is $42/50=0.840$, or 84.0%; it is the fraction of these 50 cells whose assigned labels match the supplied reference labels. (2) The union has $5+5-3=7$ cells, so the Jaccard similarity is $3/7=0.4286$. (3) Correlation is unchanged by multiplication by a positive constant: if every integrated contrast is half its reference value, the correlation can still be 1 despite magnitude attenuation. (4) A common cell type may dominate the total, so errors concentrated in a rare type can have little effect on overall accuracy; a contingency table and per-label summaries can expose that problem.