Stability Checks for Cell Embeddings

Why Check Embedding Stability?

An embedding gives each cell coordinates in a low-dimensional space, but those coordinates depend on choices such as the input cells, parameter settings, and random initialization. A visually clear arrangement in one run is not, by itself, evidence that the same local relationships would appear under reasonable alternatives. A stability check asks a narrower, testable question: when the analysis is perturbed, which nearby-cell relationships are retained?

This question complements comparisons of coordinates or plots. Rotating, reflecting, or rescaling an embedding can change coordinate values without changing which cells are neighbors. Conversely, similar-looking plots can conceal changes in a cell’s local neighborhood. Comparing neighbor sets focuses on local relationships and is invariant to rigid rotations, translations, and reflections of the coordinate system.

A stability summary is descriptive, not an inferential test. It does not estimate the probability that an embedding is “correct,” establish that a biological grouping is real, or show that one cell type causes another cell’s position. Its meaning depends on which cells, neighborhood size, distance rule, and analysis perturbations were chosen.

Define Comparable Neighborhoods

Definition: For a cell $i$, an embedding run $r$, and a specified evaluation size $k$, let $N_i^{(r)}(k)$ be the set of the $k$ other eligible cells closest to $i$ under the chosen distance in that run’s embedding. For two runs $A$ and $B$, the per-cell Jaccard similarity is $$ J_i(A,B)=\frac{|N_i^{(A)}(k)\cap N_i^{(B)}(k)|}{|N_i^{(A)}(k)\cup N_i^{(B)}(k)|}. $$ It ranges from 0 (no shared neighbors) to 1 (identical neighbor sets).

The evaluation size $k$ is a comparison choice; it need not equal a parameter used to construct an embedding. Keeping evaluation $k$ fixed makes the comparison interpretable even when the embedding settings being compared differ. Use the same distance rule for both runs, and specify how ties are handled. With $k$ neighbors in each set and no ties changing set size, the union has $2k-r$ cells if the sets share $r$ neighbors, so $J_i=r/(2k-r)$.

A related summary is the retained-neighbor fraction, $|N_i^{(A)}\cap N_i^{(B)}|/k$. It answers “what fraction of the first run’s neighbors remained in the second?” when both sets have size $k$. Jaccard similarity instead divides by the total number of distinct neighbors across both runs. State which measure is used; their numerical values are not interchangeable.

For a set $S$ of evaluated cells, the mean Jaccard similarity is

\bar{J}(A,B)=\frac{1}{|S|}\sum_{i\in S}J_i(A,B).

This treats each evaluated cell equally. It can conceal a subset of cells with very low stability, so report the distribution or useful summaries such as the median and a low quantile alongside the mean when cell-level variation matters. If the sets being compared have different sizes, do not silently apply the equal-$k$ interpretation; either choose a common evaluation size or clearly state the alternative comparison rule.

Worked Example: Calculating Jaccard Similarity

Suppose two hypothetical runs use the same cells and the same evaluation size, $k=4$. For cell A, the neighbor sets are $N_A^{(1)}=\{B,C,D,E\}$ and $N_A^{(2)}=\{B,C,F,G\}$. The intersection is $\{B,C\}$, with size 2. The union is $\{B,C,D,E,F,G\}$, with size 6.

Jaccard similarity: $J_A=2/6=0.3333$. Retained-neighbor fraction: $2/4=0.5000$. Thus half of the first run’s neighbors are retained, while one third of the distinct neighbors appearing across either run are shared.

For two additional cells, suppose their Jaccard similarities are 0.6000 and 0.3333. The three-cell mean is $(0.3333+0.6000+0.3333)/3=0.4222$. This mean describes these three cells and these two hypothetical runs; it is not a probability that the embedding is reliable.

Compare Subsamples and Parameter Settings Fairly

For parameter comparisons, use the same cells in both runs whenever possible. For subsample comparisons, a cell absent from one run has no neighborhood there, so it cannot contribute a direct paired comparison. Compare only cells present in both runs. For each such cell, form both neighborhoods from the same candidate set: the cells shared by the two subsamples, excluding the focal cell. Then find the $k$ nearest eligible cells within that shared candidate set in each embedding. This prevents a missing cell from being counted as a disagreement in one run but not the other.

There must be at least $k$ eligible shared cells for every evaluated focal cell. If this condition fails, reduce $k$ consistently or do not calculate that comparison. Because restricting candidates changes the question, state that the comparison concerns neighborhoods among shared cells. Repeat the check over several plausible subsamples rather than relying on one split. Each pairwise score remains descriptive; variation across repetitions helps show whether conclusions depend on a particular split.

The perturbation should isolate the question of interest. To examine sensitivity to an embedding parameter, hold the cell set, input representation, distance choice, and other settings fixed while changing that parameter. To examine sensitivity to cell sampling, vary the subsample while keeping the analysis choices fixed. If several ingredients change at once, a changed neighborhood cannot be attributed to any one of them.

Worked Example: Comparing Two Cell Subsamples

Consider a hypothetical comparison with six shared cells, A through F, and evaluation size $k=2$. For cell A, the first subsample’s embedding gives neighbors $\{B,C\}$, and the second gives $\{B,D\}$. The shared neighbor is B; the union is $\{B,C,D\}$, so $J_A=1/3=0.3333$.

For cell B, suppose the sets are $\{A,C\}$ and $\{A,C\}$, giving $J_B=2/2=1$. For cell C, suppose they are $\{A,D\}$ and $\{E,F\}$, giving $J_C=0/4=0$. The mean for these three focal cells is $(0.3333+1+0)/3=0.4444$.

The result indicates uneven retention: B’s two neighbors agree, C’s do not, and A’s sets overlap partly. It does not mean that 44.44% of all relationships in the full dataset are stable. The calculation uses three focal cells and candidate neighbors restricted to the six cells shared by the subsamples.

What the Overlap Measure Captures

Jaccard similarity summarizes set agreement without requiring corresponding coordinate axes. Its derivation is direct: the shared neighbors are counted once in the intersection, while every neighbor appearing in either run is counted once in the union. Dividing shared members by all distinct members gives a normalized overlap. Identical sets give numerator and denominator $k$, hence a score of 1; disjoint sets give numerator 0, hence a score of 0.

Even if two unrelated runs selected $k$ neighbors uniformly at random from the same $M-1$ possible cells, some overlap would usually occur. For any particular candidate cell, the probability it is selected in both independent sets is $(k/(M-1))^2$. Summing this probability over $M-1$ candidates gives an expected intersection size of $k^2/(M-1)$, and dividing by $k$ gives an expected retained fraction of $k/(M-1)$. This is a simple reference calculation under uniform independent selection, not a universal null model for real embeddings. Real cell neighborhoods are structured and selections are not generally uniform.

The evaluation scale matters. Small $k$ emphasizes very local relationships and may be sensitive to ties or small perturbations. Larger $k$ summarizes broader neighborhoods and can hide changes among the closest cells. It is useful to examine a small set of scientifically interpretable evaluation sizes, chosen in advance, and check whether the broad conclusion changes. Avoid choosing $k$ only because it produces a high stability score.

Worked Example: Interpreting a Parameter Comparison

Suppose a hypothetical analyst compares a baseline setting with two alternatives, using the same 100 cells and evaluation size $k=5$. Across all cells, the mean Jaccard similarities are 0.72 for alternative A and 0.49 for alternative B. For a prespecified group of 20 cells, the corresponding means are 0.68 and 0.31.

The arithmetic interpretation is that, averaged over the evaluated cells, the baseline and alternative A have greater neighbor-set overlap than the baseline and alternative B. The lower score for the 20-cell group shows that the whole-dataset mean does not describe every subset equally well. It would be reasonable to report the local relationships that persist under alternative A as more stable than those tested under alternative B, while identifying the less stable group and avoiding a claim that either setting is biologically correct.

These numbers do not establish a universal threshold for “high” stability. Whether 0.72 is adequate depends on the intended use, the plausible perturbations, the evaluation scale, and the consequences of changing neighborhoods. The analyst should also inspect per-cell scores and verify that the comparison changed only the intended parameter.

Diagnostics, Limitations, and Reporting

A single mean can hide the pattern that matters. Inspect per-cell similarities, their distribution, and whether low scores concentrate in a subset of cells or occur throughout the data. Check that the same cell identifiers are correctly matched between runs, the candidate set is identical for each paired comparison, and each focal cell has enough eligible candidates. A very high score can also be uninformative if the evaluation neighborhood is so broad that it changes little under the tested perturbations.

Coordinate-wise correlations are usually a poor substitute for neighbor overlap when embeddings can rotate, reflect, or otherwise rearrange coordinate axes. Direct coordinate comparisons require alignment and measure a different property. Neighbor overlap is also not a complete description: it does not assess global geometry, density, or distances between separated regions. The earlier material on local and global geometry across embeddings distinguishes these kinds of claims; a stability score here supports conclusions about retained local neighborhoods only.

A useful report names the cell set or subsampling scheme, the embedding settings varied, the distance rule, the evaluation $k$, the comparison candidate set, the stability summary, and notable variation across cells or repetitions. State which local relationships are retained and which are sensitive. Do not infer that stable neighborhoods prove discrete clusters, that unstable cells are errors, or that the most stable parameter choice is automatically the best scientific analysis.

Common pitfall: “The two plots look nearly identical, so the embedding is stable.” Visual similarity is subjective and can overlook changed neighbors; coordinate similarity can also be altered by rotations or reflections. Instead, define matched cell neighborhoods under a common evaluation rule, quantify their overlap, inspect cell-level variation, and limit the conclusion to the relationships actually compared.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two runs give a cell $k=3$ neighbors, with two shared cells. Find the Jaccard similarity and retained-neighbor fraction.
Why should subsample comparisons use the same candidate set for both runs?
With $M=11$ possible cells including the focal cell and $k=2$, what is the expected retained-neighbor fraction under independent uniform selection from the other cells?
A mean similarity is high, but a subset of cells has very low scores. What should be reported, and what should not be concluded?

Solutions. (1) The intersection has size 2 and the union has size $3+3-2=4$, so $J=2/4=0.5000$; the retained fraction is $2/3=0.6667$. (2) The shared candidate set makes both runs choose neighbors from the same eligible cells, so a difference does not arise merely because one run omitted a candidate. (3) There are $M-1=10$ candidates, so the expected retained fraction is $k/(M-1)=2/10=0.2000$, under the stated uniform and independent-selection assumption. (4) Report the distribution or subgroup summaries as well as the mean, and identify the affected subset. Do not claim that the embedding is uniformly stable or that low-scoring cells are erroneous; the score describes neighbor retention under the tested comparison.