Identifiability and Disentanglement in VAE Latents

Why Latent Coordinates Need Not Have a Unique Meaning

A count VAE can assign each cell a latent vector and use it to parameterize a distribution for that cell’s gene counts. It is tempting to treat each coordinate as a distinct biological quantity. But good reconstruction, or good prediction of held-out counts, does not by itself establish that coordinate 1 measures one process and coordinate 2 another. Different latent coordinates and decoder parameters can generate exactly the same distribution of observed counts.

The key issue is identifiability: whether different parameter or latent representations can be distinguished from the distribution of the observations. The generative-model and count-VAE material earlier in the course supplies the basic setup. Here the emphasis is on the symmetries that can make multiple representations observationally equivalent, and on what that equivalence means for scientific interpretation.

For latent models, the observed-data distribution is obtained by averaging over the latent variable: \(p_{\theta}(x)=\int p_{\theta}(x\mid z)p(z)\,dz\). The data do not directly reveal which coordinate system was used inside that integral. If a transformation of \(z\), together with a corresponding change to the decoder, leaves the integral unchanged, the two representations make the same observable predictions.

Rotation Invariance in a Count Model

Consider a two-dimensional latent variable \(z\sim N(0,I_2)\). For a cell with \(G\) genes, suppose that, conditional on \(z\), the counts are independent Poisson variables with log means \(\eta_g=b_g+W_g z\), where \(b_g\) is a gene-specific intercept and \(W_g\) is row \(g\) of a \(G\times2\) loading matrix \(W\). Thus \(X_g\mid z\sim\operatorname{Poisson}(\exp(\eta_g))\). This is a deliberately simple decoder used to expose the symmetry; the same argument applies to a decoder whose output parameters are generated by a neural network.

Let \(Q\) be any \(2\times2\) orthogonal matrix, so \(Q^TQ=I_2\), and define \(z'=Qz\). Since the standard Gaussian density depends on \(z\) through \(z^Tz\), this transformation preserves the prior: \(z'\sim N(0,I_2)\). If the new decoder uses \(W'=WQ^T\), then its log means satisfy \(b+W'z'=b+WQ^TQz=b+Wz\). For every original latent value and its transformed value, therefore, the conditional count distributions are identical.

The equality also follows by changing variables in the integral: an orthogonal transformation has absolute Jacobian determinant 1 and preserves \(z^Tz\), so it leaves the standard Gaussian density and integration measure unchanged. The observation model is unchanged because its log means are unchanged under the paired transformation. This is an exact equality of distributions, not merely a claim that two models happen to fit one finite data set similarly.

Equivalent Coordinates Are Not the Only Ambiguity

Rotations are an especially clear example, not a complete list of possible ambiguities. A Gaussian prior also has sign-flip and coordinate-permutation symmetries, which are special orthogonal transformations. If the decoder is sufficiently flexible, further transformations that preserve the latent prior can sometimes be absorbed into the decoder. More generally, for a one-to-one transformation \(z'=g(z)\), the transformed model can use the induced prior for \(z'\) and decoder \(p(x\mid g^{-1}(z'))\) without changing \(p(x)\). Whether such a transformed model remains inside a particular VAE’s specified family depends on that family’s restrictions.

This distinction matters. A stated prior, architecture, or constraint can remove some symmetries from the model family. But a restriction makes an interpretation identifiable only if it is justified and actually rules out observationally equivalent alternatives. A coordinate convention, such as choosing one rotation for display, can make results easier to compare without showing that the chosen axes are uniquely biological. Likewise, a fitted encoder returns coordinates in the parameterization it learned; the existence of those numerical coordinates does not prove that the observed data determine their biological meaning.

A further distinction is between identifiability and estimation. Identifiability asks whether the observable distribution uniquely determines a quantity under the model assumptions. Estimation asks how well a procedure recovers quantities from finite data. More cells can improve estimates of an identifiable feature, but do not resolve an exact symmetry: if two representations imply the same distribution for every possible data set, additional observations from that distribution cannot choose between them without extra information or restrictions.

Predictive Adequacy Is Not Biological Identification

Predictive adequacy concerns whether a model assigns appropriate probabilities to observed or held-out counts. For example, held-out log predictive density summarizes how much probability a fitted model assigns to counts not used for fitting. If two parameterizations have the same \(p(x)\), their ideal predictive performance is the same. Strong predictive performance can support the claim that the model captures useful patterns in the counts; it cannot distinguish equivalent latent coordinate systems.

Biological interpretation makes a different claim: that a latent coordinate or direction corresponds to a process such as activation of a pathway. That claim requires evidence beyond predictive fit. Relevant evidence might include a prespecified biological measurement, a perturbation with an appropriate design, or reproducible agreement with independent biological knowledge. Such evidence can help orient or validate an interpretation, but its strength depends on the study design and the assumptions used. A correlation with a marker is evidence of association, not by itself evidence that the latent factor causes the marker or represents a unique biological mechanism.

How to Diagnose and Address the Ambiguity

Identifiability is a property of the model and its assumptions, not a score that can be established by a single diagnostic plot. Still, several checks can reveal when a biological reading is especially fragile:

• Fit the same model from multiple initializations and compare both predictive performance and the learned latent directions. Similar predictions with rotated, permuted, or sign-flipped directions are consistent with equivalent solutions.
• Check whether conclusions depend on a particular coordinate, rather than on a subspace or on decoded predictions. Coordinate-specific labels that change under simple rotations should be reported cautiously.
• State which restrictions are doing interpretive work. A sparsity penalty, architectural choice, prior, or external label may favor a particular orientation, but the resulting interpretation depends on that choice and its plausibility.
• Separate validation targets. Held-out count prediction assesses predictive performance; independent biological measurements assess different claims. Neither substitutes automatically for the other.

There are several responses, with different trade-offs. One can report invariant quantities, such as the model’s count distribution or a latent subspace, when those are the features supported by the analysis. One can align representations across fits using a specified rotation or sign convention, which aids comparison but does not establish unique biology. Alternatively, one can add scientifically justified structural constraints or external information. These may improve interpretability, but they also add assumptions and can distort results if those assumptions are wrong.

In particular, independence of coordinates under the standard Gaussian prior does not establish independent biological causes. Orthogonal rotations of a standard multivariate Gaussian still have independent standard Gaussian coordinates. Likewise, a visually separated embedding or a stable optimization result is not, by itself, proof of biological disentanglement. Stability can be useful evidence about the fitting procedure, but model symmetries and scientific validation remain separate questions.

Common Mistake

Wrong reasoning: “The latent coordinate correlates strongly with a known marker, and the VAE predicts counts well, so that coordinate must be the biological process measured by the marker.” This combines a predictive claim with an interpretive claim as though one established the other.

Correction: Ask whether a different latent representation can preserve the same observed-data distribution while changing the coordinate-specific association. If so, predictive fit cannot choose between those representations. Describe the association, state the model restrictions, and seek external evidence appropriate to the biological claim. Do not interpret a chosen coordinate convention as proof of uniqueness.