Benchmarking Joint Multimodal Representations

Why Benchmark Joint Representations With More Than One Metric?

A joint representation maps cells measured in multiple modalities into a shared space or a coordinated set of coordinates. A useful representation should preserve relevant structure within each modality, support prediction across modalities when that is the task, and retain distinctions among biologically meaningful cell groups. These are related goals, but they are not interchangeable. An embedding can preserve RNA neighborhoods while distorting accessibility neighborhoods, predict a target modality well while blurring rare cell types, or separate supplied labels while emphasizing a batch effect.

Benchmarking therefore means evaluating a representation against several explicit criteria, using matched cells, consistent preprocessing, and appropriate held-out data. The earlier material on diagnosing multimodal neighborhoods and comparing separate and joint cell embeddings provides useful context for interpreting the neighborhood-based measures below. The aim here is to define complementary scores and show what each can and cannot establish.

Modality-Specific Neighborhood Preservation

Let \(n\) cells have measurements in modality \(m\), such as RNA or chromatin accessibility. For each modality, calculate a distance between cells using the modality-specific representation and define \(N_k^m(i)\) as the \(k\) nearest cells to cell \(i\), excluding \(i\). In the joint representation, let \(N_k^J(i)\) be the corresponding \(k\)-nearest-neighbor set. Fix the distance definition, feature processing, value of \(k\), and tie rule before comparing methods.

A simple neighborhood-recall score measures the fraction of each modality’s neighbors recovered in the joint representation:

This score lies between 0 and 1. A value of 1 means every cell retains all \(k\) of its modality-specific neighbors; a lower value means fewer of those neighbors are found in the joint space. Calculate \(P_m(k)\) separately for each modality. An average can summarize performance, but should not replace the modality-specific results: a strong score for one modality can hide a weak score for another.

A useful reference point follows under a simple random-neighborhood null. If, for a cell, the joint neighbor set is selected uniformly at random from the other \(n-1\) cells, each member of the modality-specific set has probability \(k/(n-1)\) of appearing in it. By adding these inclusion probabilities over the \(k\) modality-specific neighbors, the expected intersection size is \(k^2/(n-1)\); dividing by \(k\) gives an expected recall of \(k/(n-1)\). This reference assumes random sets and is not a universal null for real data, where density and batch structure can shape neighborhoods.

Cross-Modal Prediction on Held-Out Cells

A representation may also be judged by whether information available from one modality can predict measurements in another. A credible evaluation separates training and test cells before fitting any learned transformation, feature selection, or prediction rule. When cells from the same donor or experimental sample share structure, holding out entire donors or samples is often a more demanding test of generalization than a random cell split.

One transparent procedure uses a source-only representation for each test cell and a joint-space reference built from training cells. For test cell \(q\), identify its \(k\) nearest training reference cells using only information available from the source modality at prediction time. Predict a target feature by averaging its observed value among those neighbors. If the target has \(G\) features and \(\hat y_{qg}\) is the prediction for target value \(y_{qg}\), the root mean squared error (RMSE) is

The target values should be on a stated scale, such as a specified normalized or transformed expression scale; RMSE has the same units as that target scale. Smaller RMSE indicates closer predictions for the evaluated cells and features. Compare it with a simple baseline, such as predicting each target feature’s training-set mean. If joint coordinates for a test cell require its target measurements, this procedure is not a valid cross-modal prediction test: target information has leaked into the predictor. Use a modality-masked encoder or another projection that can place the query using the source modality alone.

Cell-Label Separation and Silhouette

When trusted cell labels are available, one descriptive measure of their separation in an embedding is the silhouette. For cell \(i\), let \(a_i\) be its mean distance to other cells with its own label. For each different label, calculate the mean distance from \(i\) to cells with that label; let \(b_i\) be the smallest of those means. The cell’s silhouette is

If \(a_i<b_i\), the cell is, on average, closer to its own label group than to its nearest alternative group, and \(s_i\) is positive. Values near zero indicate similar within-label and nearest-other-label distances; negative values indicate that the nearest alternative label is closer on average. The mean silhouette is the arithmetic mean of the cell-level values. It is undefined for a one-label data set or a singleton label whose within-label mean distance cannot be computed under this definition.

A Benchmarking Workflow and Its Assumptions

The measures above answer different questions and require different assumptions. Neighborhood preservation assumes that the chosen modality-specific distance and \(k\) represent a relevant local structure. Cross-modal prediction assumes that the train-test split represents the intended generalization target, that the test target was not used to construct its predictor, and that the selected loss is meaningful on the target scale. Silhouette assumes labels are available and distances are comparable within and between label groups. None of these assumptions is guaranteed by the computation itself.

A short justification explains why the random-neighborhood reference can be useful. Let \(I_j\) indicate whether a particular one of the \(k\) modality neighbors appears in a uniformly random size-\(k\) set selected from \(n-1\) possible cells. Then \(P(I_j=1)=k/(n-1)\). The intersection size is the sum of these \(k\) indicators, so linearity of expectation gives \(E[|N_k^m(i)\cap N_k^J(i)|]=\sum_{j=1}^{k}E[I_j]=k^2/(n-1)\). Dividing by \(k\) yields the stated expected recall. Independence among the indicators is not required for this expectation.

Choosing and Interpreting the Metrics

These metrics should usually be reported side by side rather than combined into an unexamined single score. If a composite is needed for a decision, its component metrics, scaling, weights, and trade-offs should be stated in advance. A high silhouette may coexist with poor within-modality neighbor preservation. A low prediction error may result from broad trends that do not resolve rare populations. Conversely, neighborhood preservation can be modest when the purpose of integration is to align related cells across modalities rather than reproduce each input space exactly.

Diagnostics include plotting score distributions across cells and samples, inspecting modality-specific results separately, checking sensitivity to \(k\), and comparing against simple baselines. For prediction, examine residuals by feature, cell type, donor, and batch; systematic errors can be hidden by an overall RMSE. For label separation, check class sizes and whether labels reflect independent biological knowledge rather than labels used to tune the representation. If labels or test outcomes were used repeatedly to select a method, the final reported performance is optimistic; use an untouched evaluation set or a nested evaluation design.

The main alternatives are not interchangeable substitutes. Neighbor overlap is direct and interpretable for local structure but depends on the chosen neighborhood scale. Prediction loss tests a task-relevant ability to transfer information, but requires a careful held-out design and a meaningful outcome scale. Silhouette is a compact label-based summary, but cannot judge unlabeled biology and may favor compact clusters even when the biology varies continuously. More elaborate metrics may be appropriate for particular goals, yet they still require explicit assumptions and controls.

Common Pitfall: Treating One Score as Biological Validation

• Wrong reasoning: “The mean silhouette is high, so the integrated embedding is biologically faithful.”
• Correction: A high silhouette shows separation for the supplied labels under the selected distance. It cannot verify label correctness, reveal unannotated biology, or rule out separation driven by batch or technical effects. Compare with modality-specific structure and held-out prediction, and inspect whether groups align with independent biological evidence.
• Another warning: A cell-level score is not automatically a sample-level inferential result. Cells from one donor share biological and technical context; uncertainty should reflect independent samples when conclusions are intended to generalize across samples.