Why Compare the Models on the Same Data?
A count VAE and a topic model can both summarize single-cell gene counts with latent variables, but they represent those counts differently. A count VAE can use a flexible decoder to model gene-specific expected counts and overdispersion. A topic model represents a cell as a mixture of gene programs and, conditional on its library size, models the allocation of counts across genes. Their latent summaries are therefore not interchangeable, and a model that gives a more attractive plot is not necessarily the better predictive model.
A useful comparison applies both models to the same cells, genes, and training–validation split. It states what was held out, how predictions were formed without using those held-out values, and which aspect of predictive performance was scored. The comparison below is deliberately small and hypothetical: it illustrates the reasoning, not evidence about an actual dataset.
What Each Model Assumes
For a count VAE, a standard formulation has a prior \(z_n\sim N(0,I)\), a variational distribution \(q_\varphi(z_n\mid x_n)\) used to approximate the latent posterior, and decoder output \(\rho_{ng}\), a gene proportion for cell \(n\). Given a library size \(\ell_n\), the decoder may set \(\mu_{ng}=\ell_n\rho_{ng}\). Under a negative-binomial likelihood, the conditional mean is \(\mu_{ng}\) and the variance is \(\mu_{ng}+\mu_{ng}^2/\theta_g\). The model assumes conditional independence of gene counts given the latent state, covariates, and modeled size information. The decoder’s flexibility does not remove the need to check whether its count distribution captures important features of the data.
In a latent Dirichlet allocation (LDA) topic model, the cell proportions \(\pi_n\) typically have a Dirichlet prior, and each topic’s gene probabilities \(\phi_k\) also has a Dirichlet prior. Conditional on \(\pi_n\), \(\phi\), and total count \(N_n\), the cell’s counts follow \(x_n\sim\operatorname{Multinomial}(N_n,\sum_k\pi_{nk}\phi_k)\). This represents counts as allocations among genes conditional on a fixed total. It assumes that each count allocation is described by the cell’s topic mixture and topic gene probabilities. The fixed-total conditioning differs from a model for the total count itself.
The comparison must respect this distinction. For example, a conditional score that fixes the held-out total assesses how well a model predicts the allocation of those counts among held-out genes; it does not assess how well the model predicts the total. The earlier material on the scVI generative model, LDA, and variational inference gives fuller treatments of the component models. Here the aim is to compare their implications on a common task.
A Shared Hypothetical Dataset and Split
Suppose the dataset contains four training cells and two validation cells measured for four genes, A, B, C, and D. The table gives hypothetical observed counts. For each validation cell, A and B are available as context, while C and D are masked for prediction.
| Cell | A | B | C | D | Role |
|---|---|---|---|---|---|
| Train 1 | 8 | 1 | 5 | 0 | Fit |
| Train 2 | 7 | 2 | 4 | 1 | Fit |
| Train 3 | 1 | 8 | 0 | 5 | Fit |
| Train 4 | 2 | 7 | 1 | 4 | Fit |
| Validation 1 | 3 | 2 | 2 | 1 | Context A, B; hold out C, D |
| Validation 2 | 2 | 3 | 0 | 3 | Context A, B; hold out C, D |
For the illustration, imagine fitting a two-dimensional count VAE and a two-topic LDA model to the training cells. The validation cells are not used to estimate the model parameters. To predict their masked genes, infer each cell’s latent summary from A and B only; do not use C or D in that inference. This separation matters: using the masked counts to infer the latent state and then scoring those same counts would make the validation prediction optimistic.
Worked Example: Reading the Two Latent Summaries
Suppose the fitted VAE maps validation cell 1 to \(z_1=(-0.8,0.2)\) and cell 2 to \(z_2=(0.6,-0.1)\). These are coordinates in the learned latent space. The decoder transforms them, along with the modeled size information, into gene-count predictions. The coordinate values are not gene-expression units, and the sign or meaning of an individual axis is not fixed.
Suppose LDA assigns cell 1 topic proportions \((0.8,0.2)\) and cell 2 proportions \((0.5,0.5)\). Each pair sums to one and describes a mixture of the two fitted topics. For example, cell 1 is more strongly assigned to topic 1 than topic 2 under this fitted model. It does not follow that the cell is biologically “80%” one cell type: topic proportions are model-based summaries of gene-count composition.
The fitted topic gene probabilities can be combined with these cell proportions to predict C and D. Assume the resulting conditional probabilities among C and D are \((0.6,0.4)\) for cell 1 and \((0.5,0.5)\) for cell 2. Suppose the VAE’s decoder, converted to the same conditional prediction target, gives \((0.25,0.75)\) and \((0.2,0.8)\), respectively. These are stipulated hypothetical fitted outputs, not values derived by training on this table.
Scoring Held-Out Gene Allocations
For a common comparison, condition on the observed total \(m_n=x_{nC}+x_{nD}\) in the masked genes and score how that total is allocated between C and D. Let \(q_n\) denote the predicted proportion for C, with \(1-q_n\) for D. Use the following binomial log score as a common scoring rule; for LDA it is a conditional predictive log score, whereas for the negative-binomial VAE it is a plug-in composition score, not the conditional allocation distribution implied by its likelihood:
For LDA, \(q_n\) comes from the topic mixture restricted and renormalized to the masked genes. For the VAE, set \(q_n=\widehat\mu_{nC}/(\widehat\mu_{nC}+\widehat\mu_{nD})\), using its predicted means, and apply the same binomial formula as a plug-in composition score. This chosen scoring rule is not generally the VAE’s conditional allocation distribution implied by its negative-binomial likelihood. The comparison is not of the models’ full count distributions: in particular, it sets aside the probability of the masked total \(m_n\).
Worked Example: Calculating the Validation Log Scores
For validation cell 1, the held-out counts are \(x_{1C}=2\) and \(x_{1D}=1\), so \(m_1=3\). The VAE predicts \(q_1=0.25\) for C. Substitution gives
The topic model predicts \(q_1=0.6\) for C. Thus
For validation cell 2, the held-out counts are \(x_{2C}=0\), \(x_{2D}=3\), and \(m_2=3\). The VAE predicts \(q_2=0.2\) for C, so its binomial score value before taking logs is \(1(0.2)^0(0.8)^3=0.512\), with log score \(\log(0.512)\approx -0.6694\). The topic model predicts \(q_2=0.5\), giving probability \(1(0.5)^0(0.5)^3=0.125\), with log score \(\log(0.125)\approx -2.079\).
Across these two validation cells, the VAE’s total plug-in composition log score is \(-1.962-0.6694=-2.631\), or \(-1.316\) per cell. The topic model’s total conditional log score is \(-0.8393-2.079=-2.919\), or \(-1.459\) per cell. Higher (less negative) log score is better under this scoring rule for these observations, so the VAE scores better on this tiny conditional-composition test. The difference is descriptive, not evidence of a reliable general advantage.
Why the Conditional Score Is Not the Whole Comparison
A fixed-total allocation score is useful because it gives both models a shared target with the held-out total fixed. But fixing that total removes one important aspect of count prediction. A count VAE may additionally provide a predictive distribution for the total, depending on its size model and likelihood. Standard LDA, as described here, conditions on the total rather than modeling it. A comparison of full joint predictions must specify a suitable total-count model for each approach and score the same joint event; it cannot be inferred from the conditional scores above.
A second issue is uncertainty. The example used one probability per model and cell. In a full analysis, latent-state uncertainty and parameter uncertainty can affect predictions. Posterior predictive probabilities should average over relevant uncertainty rather than treating a single estimated latent state as known without error. Small validation sets also make average scores unstable: a few masked counts can dominate the comparison. Repeated splits or other validation designs can show whether the ranking persists, but do not substitute for checking whether the validation task matches the intended use.
Diagnostics, Failure Modes, and Model Choice
A comparison is only informative if each model is behaving adequately. For the count VAE, inspect posterior predictive checks relevant to the application: for example, whether simulated counts reproduce library-size patterns, gene-wise means and variances, and the prevalence of zeros. A discrepancy may indicate a misspecified likelihood, size treatment, covariate structure, or decoder. Also examine whether the latent representation or predictions change substantially with reasonable modeling choices. A flexible decoder can fit complex structure, but flexibility alone does not guarantee well-calibrated predictions.
For a topic model, check whether topics are coherent enough to interpret, whether cell proportions and gene probabilities are stable across fits, and whether predictive discrepancies remain for the genes or cells of interest. Topic splitting, merging, or unstable assignments can undermine a biological interpretation even if a held-out score is competitive. These checks complement the earlier material on selecting the number of topics, topic stability, and interpreting learned topics; a numerical score does not certify that the topics are meaningful.
For both models, check the split and prediction pipeline. Confirm that held-out values did not enter training, preprocessing that estimates parameters, or latent inference for validation cells. Confirm that the score uses the same held-out genes and conditions on the same information for both methods. If one model has access to a cell’s held-out expression while the other does not, the comparison is not fair. If predictions are extreme, check for numerical errors, zero-probability events, and mismatched gene ordering before interpreting a poor score.
The models serve different priorities. A topic model is attractive when the goal is a compact mixture of gene programs, with cell proportions on a simplex and topic gene probabilities that can be inspected. A count VAE may be preferable when flexible nonlinear representations and a count likelihood are central to the task. Neither is universally superior: the relevant choice depends on the prediction target, adequacy of assumptions, interpretability needs, and performance on validation tasks that reflect intended use.
| Comparison point | Count VAE | Topic model |
|---|---|---|
| Latent summary | Continuous coordinates \(z_n\); axes need not have direct meanings | Topic proportions \(\pi_n\), with components summing to one |
| Count representation | Often gene-wise negative-binomial counts with decoder means and dispersion | Multinomial allocation across genes conditional on a total |
| Strength in this comparison | Flexible decoder and potential modeling of count variation | Explicit mixtures of interpretable gene-probability programs |
| Key caution | Latent coordinates and denoised predictions are model-dependent | Topic assignments and programs can be unstable or overly coarse |
Worked Example: Justifying a Model Choice
Suppose a team’s primary aim is to describe recurring gene programs and compare their estimated mixtures across cells. The topic model has stable, interpretable topics in the relevant training analyses, while the VAE has a slightly better conditional score on a very small validation set. The score difference alone is weak evidence, and it evaluates only allocation between C and D conditional on their total. If the topic diagnostics are satisfactory and program interpretation is the priority, choosing the topic model can be justified, while reporting its predictive limitations.
If instead the main aim is to predict gene counts and the VAE passes relevant count-based predictive checks on an adequately informative validation design, the VAE may be the more suitable choice. The decision should state which target mattered, what the validation score measured, and what diagnostics supported the choice. It should not claim that one model is generally true or that a better score proves a latent representation is biologically correct.
Common Mistakes
- Comparing unlike scores. A fixed-total or plug-in composition score is not a full count likelihood. State what is conditioned on and avoid conclusions about total-count prediction from that score.
- Letting held-out counts inform latent inference. This leaks the prediction target into the prediction process. Infer validation latents from the permitted context only.
- Reading latent coordinates as measured quantities. VAE coordinates depend on the fitted representation, while topic proportions are model-based mixtures. Neither is a direct assay measurement or a causal effect.
- Choosing by one small score difference. Tiny validation sets have limited ability to distinguish models. Consider uncertainty, stability, assumptions, and the practical objective.
- Treating a good score as a diagnostic pass. Aggregate predictive performance can hide systematic failures for particular genes or cells. Examine targeted discrepancies as well.
Check Your Understanding
Answer each question, then compare with the solutions.
- Why is it important to infer a validation cell’s latent state without using the genes held out for scoring?
- What does the conditional multinomial log score in this tutorial assess, and what does it leave out?
- For three held-out counts with \(x_C=1\), \(x_D=2\), and predicted \(q=0.4\) for C, write the probability of the observed allocation.
- Give one reason a topic model could be preferred even if a count VAE has a slightly higher validation score.
Solutions. (1) Using the held-out values in latent inference leaks the prediction target into the prediction process and can make the score optimistic. (2) It assesses the allocation between held-out genes conditional on their total; it does not assess prediction of that total. (3) The probability is \(\frac{3!}{1!2!}(0.4)^1(0.6)^2=3(0.4)(0.36)=0.432\). (4) If the aim is to describe gene programs, a topic model may be preferable when its topics are interpretable and stable; a small score advantage on one validation split need not outweigh that objective.