Latent Factor Models for High-Dimensional Profiles

Why Model Shared Variation?

A high-dimensional genomic profile may contain groups of features that vary together because of a smaller number of underlying influences. A common-factor model represents that shared variation with latent scores, while allowing each feature to retain its own residual variation. The distinction matters: the model describes a particular covariance structure, not merely a way to compress the data.

For example, a shared latent factor could summarize coordinated variation among a set of measured genes. Its score for a sample indicates that sample’s position on the latent dimension; the loadings describe how the measured features relate to that dimension. This is a statistical description of covariance. It does not establish that a factor is a biological mechanism or that it causes the observed profile.

Model and Assumptions

Definition: For sample $i$, let $x_i$ be a centered vector of $p$ feature measurements. A $q$-factor model, with $q<p$, is $$ x_i=\Lambda f_i+\epsilon_i. $$ Here $f_i$ is a $q$-dimensional vector of latent factor scores, $\Lambda$ is a $p\times q$ loading matrix, and $\epsilon_i$ is a $p$-dimensional vector of feature-specific errors. The usual covariance specification is $E(f_i)=0$, $E(\epsilon_i)=0$, $\operatorname{Cov}(f_i)=\Phi$, $\operatorname{Cov}(\epsilon_i)=\Psi$, and $\operatorname{Cov}(f_i,\epsilon_i)=0$. The unique-variance matrix $\Psi$ is diagonal, with nonnegative diagonal entries $\psi_1,\ldots,\psi_p$.

The diagonal form of $\Psi$ says that, after accounting for the common factors, the feature-specific errors are uncorrelated across distinct features. It does not say that the errors have equal variance: $\psi_j$ may differ from feature to feature. The zero cross-covariance between factors and errors ensures that the common and feature-specific parts do not contribute a covariance term with each other.

The factor covariance matrix $\Phi$ is a covariance matrix and may include correlations between factors. In the special case $\Phi=I$, the factors are uncorrelated and each has variance one. The model requires finite second moments for the covariance statements below. Normality is not required to obtain the covariance structure. If a particular inferential procedure assumes normality, that is an additional assumption, not a consequence of this model definition.

The profile measurements should be centered, or the model should include a mean vector separately. If features have very different measurement scales, the analyst must decide whether the model should describe covariance in the original units or covariance after standardization. That choice changes the interpretation of the loadings and unique variances; it is not a neutral preprocessing detail.

Deriving the Covariance Structure

The central result follows by expanding the covariance of the model. Since $x_i=\Lambda f_i+\epsilon_i$, the covariance of $x_i$ is

\begin{aligned} \Sigma &=\operatorname{Cov}(x_i)\\ &=\operatorname{Cov}(\Lambda f_i+\epsilon_i)\\ &=\Lambda\operatorname{Cov}(f_i)\Lambda^{T} +\operatorname{Cov}(\epsilon_i) +\Lambda\operatorname{Cov}(f_i,\epsilon_i) +\operatorname{Cov}(\epsilon_i,f_i)\Lambda^{T}\\ &=\Lambda\Phi\Lambda^{T}+\Psi. \end{aligned}

The final equality uses $\operatorname{Cov}(f_i,\epsilon_i)=0$ and its transpose. The common covariance $\Lambda\Phi\Lambda^T$ has rank at most $q$; the diagonal matrix $\Psi$ adds feature-specific variance. Thus the model is not claiming that the entire covariance matrix has low rank. Instead, it claims that the off-diagonal covariance is generated by a low-dimensional common component, while the remaining variance is feature-specific.

For features $j$ and $k$, the covariance is

\operatorname{Cov}(x_{ij},x_{ik}) =\lambda_j^{T}\Phi\lambda_k+\begin{cases} \psi_j,&j=k,\\ 0,&j\ne k, \end{cases}

where $\lambda_j^T$ is row $j$ of $\Lambda$. For a single factor with variance one, this becomes $\operatorname{Cov}(x_{ij},x_{ik})=\lambda_j\lambda_k$ for distinct features. Features with loadings of the same sign tend to have positive model covariance through that factor; opposite signs contribute negative covariance. This is a statement about the factor component, not a guarantee about every observed pair in a finite data set.

Worked Example: A One-Factor Covariance Matrix

Suppose two hypothetical centered genomic measurements follow a one-factor model with $\Phi=1$, loadings $\lambda_1=2$ and $\lambda_2=-1$, and unique variances $\psi_1=1$ and $\psi_2=4$. These values are hypothetical model parameters, not results from a real study.

The loading matrix is $\Lambda=(2,-1)^T$, so the common covariance is $\Lambda\Lambda^T=\begin{pmatrix}4&-2\\-2&1\end{pmatrix}$. Adding $\Psi=\begin{pmatrix}1&0\\0&4\end{pmatrix}$ gives

\Sigma=\Lambda\Lambda^T+\Psi =\begin{pmatrix}4&-2\\-2&1\end{pmatrix} +\begin{pmatrix}1&0\\0&4\end{pmatrix} =\begin{pmatrix}5&-2\\-2&5\end{pmatrix}.

The model variances are $5$ and $5$, in the squared units of their respective measurements, and their covariance is $-2$ in the product of those units. Their model correlation is $-2/\sqrt{5\cdot5}=-0.4000$. The negative covariance arises from opposite-signed loadings; the two different unique variances affect the diagonal, not the off-diagonal covariance.

Loadings, Scores, and Variance Contributions

A loading describes how a feature participates in the common-factor part of the model. When factors are uncorrelated with unit variance, $\Phi=I$, the common variance for feature $j$ is the sum of its squared loadings, $\sum_{r=1}^{q}\lambda_{jr}^{2}$. Its total model variance is that sum plus $\psi_j$. The proportion of its variance attributed to common factors is therefore

\frac{\lambda_j^T\Phi\lambda_j} {\lambda_j^T\Phi\lambda_j+\psi_j}.

When $\Phi=I$, the numerator reduces to the sum of squared loadings. When factors are correlated, the numerator also includes terms involving those correlations, so simply summing squared loadings is not generally correct.

The score $f_i$ is latent: it is not directly recorded as one of the profile measurements. In an application, estimated scores are used to summarize samples, but the model statement itself distinguishes the unobserved score from the observed vector $x_i$. A loading is not a sample score, and a large loading alone does not imply that every sample has a high factor score.

Worked Example: Two Factors and Three Features

Consider a hypothetical three-feature profile with two uncorrelated, unit-variance factors, so $\Phi=I$. Let $\Lambda=\begin{pmatrix}1&0.5\\0.5&1\\1&-1\end{pmatrix}$ and $\Psi=\operatorname{diag}(0.25,0.5,1)$. The shared covariance is found by taking row-by-row inner products of $\Lambda$.

The common variances are $1^2+0.5^2=1.25$, $0.5^2+1^2=1.25$, and $1^2+(-1)^2=2$. The common covariances are $1(0.5)+0.5(1)=1$ for features 1 and 2, $1(1)+0.5(-1)=0.5$ for features 1 and 3, and $0.5(1)+1(-1)=-0.5$ for features 2 and 3. Thus

\Lambda\Lambda^T =\begin{pmatrix}1.25&1&0.5\\1&1.25&-0.5\\0.5&-0.5&2\end{pmatrix}, \qquad \Sigma=\Lambda\Lambda^T+\Psi =\begin{pmatrix}1.5&1&0.5\\1&1.75&-0.5\\0.5&-0.5&3\end{pmatrix}.

For feature 1, the common variance is $1.25$ and the total variance is $1.25+0.25=1.5$, so its common-variance proportion is $1.25/1.5=0.8333$. For features 1 and 2, the model correlation is $1/\sqrt{1.5\cdot1.75}=0.6172$. The negative covariance of $-0.5$ between features 2 and 3 results from their loading patterns, while the unique variances increase only the diagonal entries.

How This Covariance Model Differs From PCA

PCA describes directions that account for variance in the observed covariance matrix. Its component scores are projections of the observed profiles onto selected directions, and a rank-$q$ PCA approximation retains selected components of the total covariance. The common-factor model instead partitions covariance into a low-rank shared part and a diagonal unique-variance part. These are different assumptions about what remains after representing shared variation.

For a numerical comparison, use the two-feature covariance matrix $\Sigma=\begin{pmatrix}2&2\\2&5\end{pmatrix}$. It has a one-factor representation with $\Lambda=(1,2)^T$, $\Phi=1$, and $\Psi=I$. The common part is $\begin{pmatrix}1&2\\2&4\end{pmatrix}$, and each feature has unique variance $1$. The eigenvalues of $\Sigma$ are $6$ and $1$: its trace is $7$, and the discriminant is $\sqrt{(2-5)^2+4(2)^2}=5$.

The leading PCA component keeps variance $6$, leaving a rank-one residual covariance matrix with eigenvalue $1$. Its residual matrix is $\frac{1}{5}\begin{pmatrix}4&-2\\-2&1\end{pmatrix}$, which has off-diagonal entries $-0.4$. By contrast, the factor model’s feature-specific covariance is $I$, with zero off-diagonal entries. PCA’s residual covariance in this example is not the factor model’s diagonal unique-variance matrix. PCA and factor models can produce similar summaries, but one should not interpret a PCA residual automatically as feature-specific independent noise.

Aspect	Common-factor model	PCA
Representation	Shared low-rank covariance plus diagonal unique variance	Selected directions approximate total observed covariance
Scores	Latent variables in a covariance model	Projections of observed profiles onto component directions
Residual structure	Feature-specific errors are uncorrelated across features under the model	Discarded components need not imply diagonal residual covariance
Interpretive aim	Describe common covariance while retaining feature-specific variance	Summarize variance in a smaller number of directions

Assumptions, Diagnostics, and Limitations

The central structural assumption is that off-diagonal covariance can be represented by $\Lambda\Phi\Lambda^T$. The diagonal $\Psi$ assumption is also substantive: unexplained residual associations between features are not allowed by this model. In practice, a fitted model can be assessed by comparing its implied covariance with the empirical covariance, especially the residual off-diagonal entries. Large, systematic residual correlations suggest that the chosen factors or the diagonal-error structure do not adequately represent the data.

Other warning signs include estimates of unique variances near or below zero in an unconstrained fitting procedure, highly unstable loadings under small data changes, or factors whose interpretation depends heavily on a few measurements. These problems may reflect too few or too many factors, strong outliers, preprocessing choices, or covariance structure the model cannot express. They are reasons to investigate the fit, not automatic proof of a particular alternative explanation.

High-dimensional data add practical constraints. With $p$ features and fewer samples than features, the empirical covariance matrix is singular, so it cannot be treated as a well-behaved full-rank estimate without additional choices. The factor model’s structure can be more economical than an unrestricted covariance matrix, but it does not remove the need to justify the number of factors or assess stability. The model is also not uniquely parameterized without constraints: different factor coordinate systems can describe the same common covariance. Loadings should therefore be interpreted with care, particularly when comparing separate fits.

Common Mistake

Pitfall: “A factor model is just PCA with a different name, so the discarded PCA variance is the unique variance.” This reasoning is incorrect. PCA decomposes total variation into orthogonal component directions; a common-factor model specifies a diagonal feature-specific covariance in addition to shared covariance. The PCA remainder can contain off-diagonal covariances, so it need not equal $\Psi$.

The correction is to state the model being used and inspect its implied covariance. In a factor model, $\Psi$ has a precise role: its diagonal entries are feature-specific variances, and its off-diagonal entries are zero. A PCA reconstruction residual is defined by the components retained and discarded; it is not automatically a collection of independent feature errors.

Worked Example: Checking a Proposed Covariance

Suppose a hypothetical two-feature model has one factor with variance $1$, loadings $(1,2)^T$, and proposed unique variances $0.5$ and $1$. The implied shared covariance is $\begin{pmatrix}1&2\\2&4\end{pmatrix}$, while $\Psi=\begin{pmatrix}0.5&0\\0&1\end{pmatrix}$. Therefore,

\Sigma=\begin{pmatrix}1.5&2\\2&5\end{pmatrix}.

The implied feature variances are $1+0.5=1.5$ and $4+1=5$, and the implied covariance is $1(2)=2$. The correlation is $2/\sqrt{1.5\cdot5}=2/\sqrt{7.5}=0.7303$. A proposed covariance with these diagonal entries but covariance $1$, for instance, would not match this specified model: its off-diagonal entry must equal $2$. The calculation checks model implications; it does not show that the model fits observed data.

Check Your Understanding

Answer each question, then compare with the concise solutions below.

If factor scores and errors have zero means, but their cross-covariance is nonzero, which step in the covariance derivation changes?
For one unit-variance factor with loadings $3$ and $-2$, what is the model covariance between the two distinct features?
A feature has loading vector $(1,2)$, $\Phi=I$, and unique variance $2$. What are its common variance, total variance, and common-variance proportion?
Why does a diagonal $\Psi$ not imply equal residual variance for every feature?
What key distinction prevents interpreting the residual covariance from a truncated PCA as $\Psi$ automatically?

Solutions. 1. The cross terms $\Lambda\operatorname{Cov}(f_i,\epsilon_i)$ and $\operatorname{Cov}(\epsilon_i,f_i)\Lambda^T$ remain, so $\Sigma=\Lambda\Phi\Lambda^T+\Psi$ no longer follows as written. 2. The covariance is $3(-2)=-6$. 3. The common variance is $1^2+2^2=5$; total variance is $5+2=7$; the common-variance proportion is $5/7=0.7143$. 4. Diagonal means cross-feature error covariances are zero; the diagonal entries $\psi_j$ can still differ. 5. PCA approximates total covariance by retained directions, whereas a factor model specifies shared covariance plus diagonal feature-specific covariance; the PCA remainder need not be diagonal.