Probabilistic PCA and Its Noise Model

From a Geometric Decomposition to a Probability Model

Principal component analysis (PCA) describes directions of variation in a collection of profiles. Probabilistic PCA (PPCA) gives a probability model for that variation: each profile is generated from a lower-dimensional latent score plus feature-level noise. This formulation makes the assumptions behind the low-rank representation explicit and provides a likelihood for assessing how well the model describes profiles.

Suppose each observation is a profile of \(d\) features, such as measurements on \(d\) genes. Write the profile for observation \(i\) as \(x_i\), a \(d\)-dimensional vector. PPCA uses \(q\) latent variables per observation, where \(q<d\) is treated as fixed here. The matrix \(W\) maps latent scores into feature space; its \(d\) rows correspond to features.

The Gaussian distributions and independence assumptions matter: they determine both the profile distribution and the likelihood used to fit the model. The latent scores have mean zero and unit variance in each direction, while \(\mu\) accounts for the feature means. Conditional on a latent score, the remaining errors have mean zero and covariance \(\sigma^2I_d\). Thus, the model does not allow a gene-specific noise variance or residual correlation between different genes.

The Implied Covariance and the Meaning of Isotropic Noise

Because a linear transformation of Gaussian variables is Gaussian, the model gives a multivariate normal distribution for each profile. Taking expectations gives \(E(x_i)=\mu\). For the covariance, independence makes the cross-covariance terms zero:

The matrix \(WW^T\) is positive semidefinite and has rank at most \(q\); it describes covariance represented through the latent scores. The term \(\sigma^2I_d\) adds the same variance to every feature and zero covariance between distinct features. Consequently,

Isotropic means that the noise covariance is the same in every direction in feature space: for any unit vector \(a\), the noise projection \(a^T\epsilon_i\) has variance \(\sigma^2\). In the original feature coordinates, this entails equal residual variances and zero residual covariances. It does not mean that the observed profiles have equal variances or are uncorrelated. The low-rank term \(WW^T\) can create different marginal variances and correlations across features.

The noise assumption concerns variation left after accounting for latent structure. For genomic profiles, this may be a useful approximation when residual variation is roughly comparable across the analyzed features. It can be strained when genes have markedly different measurement precision, different residual variability, or remaining correlated technical effects. If profiles have been standardized, \(\sigma^2\) is in squared standardized units; if measured in original units, it is in squared units of the corresponding feature measurements. Standardization changes the substantive interpretation of equal noise and should be considered before fitting.

Latent Scores Given an Observed Profile

The model also gives a distribution for the latent score conditional on an observed profile. For \(x_i\), let \(r_i=x_i-\mu\). Combining the standard normal density for \(z_i\) with the Gaussian density for \(r_i\) given \(z_i\), the terms involving \(z_i\) in the log density are proportional to $$ -\frac{1}{2}\left[z_i^Tz_i+\frac{(r_i-Wz_i)^T(r_i-Wz_i)}{\sigma^2}\right]. $$ Collecting the quadratic and linear terms in \(z_i\), and completing the square, gives

The conditional mean is a model-based estimate of the latent score, and the conditional covariance represents uncertainty about that score. Both depend on the fitted loadings and noise variance. A score is therefore not treated as an error-free observed quantity: uncertainty is larger when the profile contains little information about its latent coordinates relative to the noise.

Fitting PPCA and Its Connection to PCA

For independent profiles under the Gaussian model, fitting by maximum likelihood amounts to fitting the covariance \(WW^T+\sigma^2I_d\), along with the mean. To describe the covariance fit, let \(\bar{x}\) be the sample mean and define the maximum-likelihood sample covariance using divisor \(n\): $$ S=\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})(x_i-\bar{x})^T. $$ Let its eigenvalues be \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_d\geq0\), with corresponding unit eigenvectors \(u_1,\ldots,u_d\). The following interior solution applies for a fixed \(q<d\), when the residual variance is positive and the leading eigenvalues exceed it.

A short justification follows from the covariance likelihood. For a candidate covariance \(C\), the part of the negative twice log likelihood that depends on \(C\), apart from a positive constant factor, is \(\log|C|+\operatorname{tr}(SC^{-1})\). The best-fitting covariance aligns its directions with those of \(S\); in the \(q\) directions allowed to have extra variance, the fitted eigenvalues match the corresponding sample eigenvalues. In each remaining direction the model requires the same eigenvalue, \(\sigma^2\). If that shared value is \(v\), the terms involving the remaining eigenvalues are $$ (d-q)\log(v)+\frac{\sum_{j=q+1}^{d}\lambda_j}{v}. $$ Differentiating with respect to \(v\) and setting the derivative to zero yields $$ \frac{d-q}{v}-\frac{\sum_{j=q+1}^{d}\lambda_j}{v^2}=0 \quad\Longrightarrow\quad v=\frac{1}{d-q}\sum_{j=q+1}^{d}\lambda_j. $$ In the leading directions, \(WW^T\) must supply the excess over the common noise eigenvalue, so the loading magnitudes are \(\sqrt{\lambda_j-\hat{\sigma}^2}\). This gives the stated representative for \(\hat W\). If the estimated residual variance is zero or the required excesses are not positive, this interior formula does not describe a regular positive-noise solution.

Diagnostics, Alternatives, and Limitations

The eigenvalue calculation makes clear what PPCA gains and what it constrains. Ordinary PCA gives a low-rank geometric summary but does not, by itself, specify a probability distribution for residuals. PPCA supplies a Gaussian likelihood and a common residual variance. A factor model with feature-specific uniquenesses relaxes the equal-residual-variance constraint, but uses a more flexible residual covariance structure. Neither model is automatically appropriate: the choice depends on whether its assumptions describe the profile measurements well.

Useful diagnostics focus on what remains after fitting the latent structure. Examine residual variance by feature and residual correlations across features; systematic differences or persistent correlations challenge \(\sigma^2I_d\). Also inspect unusually large residuals and patterns associated with batches or other technical variables. Such patterns may indicate that a low-rank Gaussian model with isotropic noise is missing structure. These diagnostics are warnings about model fit, not proof of a particular biological mechanism.

A common mistake is to interpret isotropic noise as saying that all genes have the same total variance or that observed genes are independent. That is incorrect: the assumption says the residual noise covariance, conditional on the latent score, is \(\sigma^2I_d\). The marginal covariance is \(WW^T+\sigma^2I_d\), which can have unequal variances and substantial correlation. A second pitfall is to read \(\sigma^2\) as a universal biological variability parameter. It is a model-based residual variance in the analyzed feature scale; preprocessing, measurement technology, and omitted structure all affect its interpretation.

PPCA is useful when a probabilistic low-dimensional representation is desired and a shared residual variance is a defensible approximation. It is limited when residual variation is strongly feature-dependent, non-Gaussian, or structured by effects outside the latent representation. The fixed \(q\) is an input to the fit; this covariance derivation does not determine which value should be used.