Why Topic Stability Requires Alignment
A topic model can return the same collection of gene programs under different topic labels. For example, topic 1 in one fitted model might correspond to topic 3 in another. Comparing topics by their numeric labels would make these equivalent solutions look different. Assessing stability therefore begins by matching topics across fitted models, then summarizing how similar the matched gene programs are.
This tutorial treats the number of topics \(K\) as fixed, as selected in the earlier material on choosing the number of topics. The aim is not to decide whether a topic is biologically meaningful, but to ask whether similar gene-weight patterns recur when the model is refitted from another random initialization or trained on a perturbed set of cells.
The gene set and its ordering must be consistent across runs. If runs use different gene sets, first restrict each topic to a common set of genes and renormalize its weights to sum to one. Comparisons also require the same topic representation: normalized gene-weight vectors should not be compared directly with unnormalized scores whose scales differ arbitrarily.
Similarity and One-to-One Matching
A standard choice for comparing two nonnegative topic vectors is cosine similarity. For nonzero vectors \(u\) and \(v\), cosine similarity measures the angle between them, ignoring differences in their overall magnitudes. For normalized topic vectors it lies between zero and one.
A value near one means the vectors point in similar directions; a value near zero means their weights emphasize different genes. Cosine similarity compares the full vectors, so it can be affected by many low-weight genes. Alternatives include overlap among the top-ranked genes, rank correlation, or a distance based on probability distributions. These emphasize different features and can produce different matches. The similarity measure should be specified before comparing results.
For two runs, calculate every cross-run similarity to form a \(K\)-by-\(K\) matrix. A one-to-one matching chooses a permutation of the topics in the second run that maximizes the sum of matched similarities. This is a linear assignment problem, commonly solved by the Hungarian algorithm. One-to-one matching is appropriate when both runs have the same \(K\) and each topic is expected to have one counterpart; it does not force the counterpart to be a good match.
The assignment maximizes total similarity, but the mean alone is incomplete. It can hide an individual topic with a poor match, or a near-tie in which one topic has several plausible counterparts. Report the matched similarities individually, or at least alongside their minimum, median, and spread. A score is descriptive of the chosen runs and metric; it is not a p-value, a probability that a topic is true, or a universal cutoff for adequacy.
Why Permutation Matching Is Necessary
Topic labels are arbitrary because permuting topic indices does not change the fitted collection of gene-weight vectors. If a model is represented by \((\phi_1,\ldots,\phi_K)\), relabeling it as \((\phi_{\sigma(1)},\ldots,\phi_{\sigma(K)})\) for any permutation \(\sigma\) leaves the collection unchanged. Consequently, a label-by-label comparison can report low agreement even when the two collections are identical.
The assignment objective has a direct justification. For any fixed pairing \(\pi\), the average cosine dissimilarity of its matched pairs is \(K^{-1}\sum_k(1-S_{k,\pi(k)})\). Since each pairing contains exactly \(K\) terms,
Thus, minimizing average cosine dissimilarity is exactly equivalent to maximizing average similarity. Searching over permutations handles arbitrary topic labels without changing the similarity criterion. This justification depends on comparing the same number of topics and using a one-to-one assignment; it does not establish that one-to-one matching is adequate when topics split or merge.
Worked Example: Matching Topics Across Two Initializations
Suppose two hypothetical fits each have three topics. Their cosine similarities, with rows for the first fit and columns for the second, are
For one pair, suppose the first fit's topic vector is \(u=(0.6,0.3,0.1)\) and the second fit's candidate topic vector is \(v=(0.5,0.4,0.1)\), over three genes. Their cosine similarity is
This agrees with the matrix entry rounded to 0.98. The diagonal assignment has total similarity \(0.98+0.91+0.95=2.84\). For comparison, swapping the first two topics while keeping the third matched gives \(0.31+0.27+0.95=1.53\). The diagonal is the optimal assignment here. Its mean matched similarity is \(2.84/3=0.9467\). The values support strong agreement for these three matched topics under this metric, but do not show that the topics are biologically valid.
Comparing Initializations and Resampled Cells
To assess sensitivity to random initialization, fit the same model specification multiple times to the same cell data, using different random starts. Match every pair of runs, or select a reference run and match all other runs to it. Pairwise comparison avoids relying on one potentially atypical reference, although it creates many scores that are not independent because runs are reused across pairs. A useful summary includes the distribution of matched similarities and identifies topics whose matched scores are consistently low.
To assess sensitivity to which cells were observed, fit the model on repeated subsamples or bootstrap resamples and compare the resulting topics on a common gene set. Resampling cells changes the data, so this assesses robustness to sampling perturbation in addition to optimization. For a bootstrap that duplicates cells, duplicates are part of the resampled training data; the comparison is still between the fitted gene-weight vectors. State the resampling design, the proportion of cells retained, and whether model settings were held fixed.
These two perturbations answer different questions. Repeated starts ask whether the optimization procedure returns similar solutions on the same input. Resampling asks whether solutions persist under changes to the input cells. If both are used, summarize them separately rather than combining their scores as though they represented one source of variation.
Worked Example: Summarizing Stability Across Cell Subsamples
In a hypothetical analysis, one reference fit is compared with four fits trained on different cell subsamples. After matching, a particular topic has cosine similarities \(0.88\), \(0.84\), \(0.91\), and \(0.47\). The mean is
The sorted scores are \(0.47, 0.84, 0.88, 0.91\), so the median is \((0.84+0.88)/2=0.86\). Three of four scores are at least 0.80, giving a descriptive proportion of \(3/4=0.75\). The high median but lower mean and minimum show that this topic usually matches well in these resamples, but has one notably weak match. The value 0.80 is only a descriptive threshold in this example, not a generally validated decision rule. The weak result should prompt inspection of that resample and the full matching, not an automatic declaration that the topic is unstable.
Diagnosing Ambiguous Matches and Failure Modes
A matching algorithm always returns an assignment when the number of topics is equal, even if the solutions differ substantially. Therefore, inspect more than the assignment itself. Useful diagnostics include the distribution of matched similarities, the lowest-scoring pairs, and the margin between a topic's best and second-best candidate in the other run. A small margin indicates ambiguity, even when the selected match has a reasonably high score.
A one-to-one assignment can also conceal topic splitting or merging. One topic in a run may resemble two topics in another run, or two topics may both resemble the same topic. The algorithm must assign each topic only once, so one plausible relationship will be omitted. Review the full similarity matrix for such patterns. If the number of topics differs, a forced one-to-one assignment is not directly defined for all topics; use an explicitly stated strategy that allows unmatched topics, or compare only under a clearly justified common subset. Do not interpret a forced match as evidence of correspondence.
Worked Example: A High Match That Hides a Split
Suppose a hypothetical reference fit has two topics and another fit has three. The cross-fit similarity matrix is
A one-to-one assignment over the two reference topics can pair the first reference topic with the first candidate (\(0.86\)) and the second reference topic with the third candidate (\(0.90\)), for a mean of \((0.86+0.90)/2=0.88\). Yet the first reference topic also has a very similar candidate at \(0.83\). Its top-two margin is only \(0.86-0.83=0.03\), while the third candidate is a poor match for that reference topic. The two similar candidates could reflect a split or redundancy; the matrix alone does not decide which explanation is correct. Reporting only the mean of 0.88 would conceal this ambiguity. Because the fits have different topic counts, the calculation is a partial matching, not evidence that all topics correspond one-to-one.
Other common failure modes include comparing differently normalized weights, silently changing the gene set, and relying on a similarity metric that rewards only the most prominent genes when the intended question concerns the whole distribution. Numerical optimization that has not converged can also make fitted topics vary across starts. Stability scores diagnose variation in the fitted outputs; they do not by themselves identify its cause.
Use the same topic count when evaluating initialization stability, and specify which cell perturbations are used for resampling stability.
Use the same gene set, gene order, and normalization for every fitted run being compared.
Calculate all cross-run similarities using the same definition, rather than selecting a metric after seeing which one gives the strongest agreement.
Optimize the one-to-one assignment when its assumptions fit, and inspect low scores, small margins, and possible split or merge patterns.
Describe stability across the particular starts or resamples examined; do not treat it as proof of biological validity or generalizability.
Choosing Among Stability Summaries
| Approach | What it emphasizes | Main caution |
|---|---|---|
| Cosine similarity on full topic vectors | Overall alignment of normalized gene weights | Can be influenced by many low-weight genes and by the chosen feature set |
| Top-gene overlap | Agreement among the most highly weighted genes | Depends on the chosen list length and treats genes inside the list similarly |
| Rank correlation | Similarity in ordering of gene weights | Can downplay differences in the magnitude of weights |
| Reference-run matching | Simple comparison of all runs to one fitted solution | Conclusions can depend on the reference run |
| All-pairs matching | Agreement across the full set of fitted runs | Pairwise scores reuse runs and should not be treated as independent observations |
No single summary is best for every scientific aim. A strong practice is to choose a primary metric in advance and use a complementary summary as a sensitivity check. If the conclusions change substantially with reasonable metrics, that disagreement is itself informative: stability depends on which aspect of the gene-weight pattern is being assessed.
Common Mistake and Interpretation Limits
- Wrong reasoning: “The mean matched similarity is high, so the topics are real gene programs.” Correction: The score shows similarity among fitted gene-weight vectors under a particular metric and perturbation scheme. It does not establish biological validity.
- Wrong reasoning: “Topic 1 changed across runs, so the model is unstable.” Correction: Topic labels are arbitrary. Align the topics before comparing their vectors.
- Wrong reasoning: “The assignment returned a match, so that match is good.” Correction: A one-to-one algorithm returns an assignment even when all candidate similarities are weak. Examine the scores and alternatives.
- Wrong reasoning: “The resampling stability score is the same as initialization stability.” Correction: Resampling changes the input cells; repeated starts change the optimization initialization. Report these sources of variation separately.
Stability also depends on modeling choices such as the gene set, preprocessing, priors, and optimization settings. Holding these choices fixed isolates the perturbation being studied, but it does not show robustness to the choices that were held fixed. Stability is one diagnostic of reproducibility within the specified analysis, not a substitute for other checks or substantive interpretation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why can a label-by-label comparison give a misleading stability result when two fits contain the same topics in a different order?
- Two fitted runs have four topics. Their matched similarities are \(0.90\), \(0.88\), \(0.86\), and \(0.30\). Calculate the mean and explain what information it hides.
- What does a small difference between a topic's best and second-best cross-run similarity suggest?
- What is the difference between stability across random starts and stability across resampled cells?
- Does a high matched cosine similarity establish that a gene program is biologically valid? Explain.
Solutions. (1) Topic indices are arbitrary, so corresponding gene-weight vectors may have different labels; matching removes this label permutation before comparison. (2) The mean is \((0.90+0.88+0.86+0.30)/4=2.94/4=0.735\). It summarizes all four pairs but can conceal that three are high and one is much lower; report the individual scores or their spread as well. (3) The match is ambiguous because another candidate is nearly as similar, potentially indicating redundant topics or a split/merge pattern. (4) Random-start comparisons assess sensitivity to optimization initialization on the same input data; cell-resampling comparisons assess sensitivity to changes in the cells used for fitting. (5) No. It establishes similarity of fitted vectors under the chosen metric and comparison design, not biological validity or generalizability.