Choosing the Number of Topics

Why Topic Count Is a Modeling Choice

A topic model represents each cell as a mixture of gene programs, while each program assigns weights to genes. Choosing the number of topics, \(K\), therefore determines how many programs the model can use to describe variation in the count matrix. Too few topics can force distinct patterns into the same program; too many can split one coherent pattern into near-duplicates or fit details that do not predict well beyond the data used for fitting.

No single statistic identifies a universally correct \(K\). Held-out predictive fit asks how well a fitted model predicts counts not used to estimate it. Coherence asks whether genes grouped in a topic tend to co-occur in cells. Redundancy asks whether different topics appear to represent nearly the same gene pattern. These address different properties, so selection should compare them rather than collapse them uncritically into one score.

Constructing a Fair Predictive Comparison

For count data, a useful evaluation is document completion: divide each cell's observed counts into context counts used to infer that cell's topic mixture, and target counts reserved for scoring. The topic distributions are fitted using training cells. For a held-out cell, the model uses its context counts to learn about its mixture, then evaluates the conditional predictive probability of its target counts. This assesses whether the topics learned in training can predict additional counts in a cell with partial information.

Let \(x_c\) denote context counts and \(y_c\) target counts for held-out cell \(c\). The predictive score for \(K\) topics can be written as a sum of conditional log probabilities. If there are \(M\) target count units across the held-out cells, report the score per target count unit so comparisons remain interpretable across splits:

A larger (less negative) \(S_K\) is better under this log-score convention. The conditional probability generally averages over uncertainty in the held-out cell's topic mixture given its context counts; using the target counts to infer the mixture before scoring those same target counts would give an overly optimistic evaluation. In practice, the probability may be approximated by an inference procedure, so the approximation method and computation should be kept comparable across candidate values of \(K\).

A fair comparison uses the same training cells, held-out cells, gene set, context/target split, and scoring convention for every \(K\). If the goal is prediction for new biological samples, the split should respect that goal: holding out cells from samples already represented in training mainly evaluates prediction within those samples, not generalization to unseen samples. Counts, library-size handling, and preprocessing must also be defined consistently.

Coherence and Redundancy as Complementary Diagnostics

A common coherence measure uses normalized pointwise mutual information (NPMI) for pairs among a topic's top-weighted genes. Define \(p(g)\) as the fraction of cells in a chosen reference set in which gene \(g\) is detected, and \(p(g,h)\) as the fraction in which both genes are detected. For a pair with \(0<p(g,h)<1\), define

For pairs in this defined range, NPMI is between \(-1\) and \(1\). Positive values indicate co-detection more often than expected under independence, zero is the independence benchmark, and negative values indicate less co-detection than expected. These interpretations apply to the detection events in the selected reference cells; they do not establish a regulatory relationship or a causal interaction.

The boundary cases require explicit handling. If \(p(g,h)=0\), the logarithm is undefined; if \(p(g,h)=1\), the denominator is zero and, because both marginal probabilities are then 1, the numerator is also zero. A reproducible convention is to exclude both types of pairs from the NPMI average and report how many eligible pairs remain. Do not assign them an ordinary NPMI value. If no eligible pairs remain, the topic's average NPMI is undefined rather than zero. Results can also depend on the number of top genes chosen and on the reference cells, so those choices should be fixed across candidate topic counts.

Redundancy can be assessed by comparing topic gene-weight vectors. For example, cosine similarity between two nonzero weight vectors \(w_k\) and \(w_l\) is their dot product divided by the product of their Euclidean lengths. Values close to 1 indicate similar directions in gene-weight space. Similarity is a diagnostic, not a hypothesis test: a threshold such as 0.85 is a practical flag chosen for a particular analysis, not a universal boundary between distinct and redundant biology.

A Selection Rule With Multiple Criteria

The choice should be made using a declared comparison rule, not by searching for a single magically decisive number. One practical approach is to identify a range of topic counts whose predictive scores are close to the best score, then prefer a simpler model within that range if its topics are at least as coherent and less redundant. “Close” should be defined in advance in the scale being used—for example, a tolerance in nats per held-out count unit—and treated as a practical decision threshold, not a significance test.

The reasoning behind held-out scoring is straightforward. Fitting can adapt a model to training observations, so a training score alone does not measure how well the fitted model predicts unused observations. A held-out score evaluates predictions on data not used to estimate the topic distributions. Its average estimates predictive performance for observations generated under a comparable sampling and preprocessing scheme. The score does not directly reward interpretability, and it can favor added topics when they improve prediction, even if the new topics are difficult to distinguish from existing ones.

Assumptions, Diagnostics, and Alternatives

The comparison is meaningful only if candidate models are evaluated on genuinely comparable data and with a consistent likelihood and inference procedure. The cells used for evaluation should represent the prediction target, and the held-out counts must not have been used to fit the topic distributions. Since stochastic inference and finite evaluation samples can make scores vary, small score differences should not be interpreted as decisive without checking that they are practically meaningful and not driven by a few cells or genes.

Several diagnostics help detect a misleading selection. Confirm that each \(K\) was scored on the same target counts and with the same normalization or exposure treatment. Examine whether conclusions change substantially under reasonable alternative held-out splits or choices of top genes. Record the number of NPMI pairs excluded because \(p(g,h)=0\) or \(p(g,h)=1\); a mean based on very few eligible pairs is less informative. Inspect highly similar topic pairs directly, since a similarity threshold can flag overlap but cannot decide whether two programs are scientifically distinct.

Document completion is not the only predictive design. One may instead hold out whole cells and evaluate their predictive probability under the model without using their counts to infer their cell-specific mixture. That design targets new-cell prediction but can be more demanding and differs from completion, so scores from the two designs should not be compared as if they were the same quantity. Training likelihood is a weaker selection tool: it usually improves as flexibility is added and therefore does not by itself reveal whether added topics generalize. Coherence alone is also insufficient; common, broadly detected genes can co-occur for reasons unrelated to a specific program, while rare genes may produce unstable pair frequencies.