Common-Factor Model and Its Covariance Decomposition

Why a Common-Factor Model?

When several measured variables move together, one explanation is that they share influences that are not directly observed. A common-factor model expresses this idea with a small vector of latent factors: each observed variable responds to those shared factors and also has its own specific variation. The model’s central covariance statement separates these two sources of variation.

Let \(X\) be a \(p\)-dimensional vector of observed variables and let \(F\) be a \(q\)-dimensional vector of latent factors. In the standard model, \(q\) is usually smaller than \(p\). The \(p\)-by-\(q\) loading matrix \(\Lambda\) describes how the factors enter the observed variables. A separate error vector represents variation specific to the observed variables.

Definition: The Gaussian common-factor model is \(X=\mu+\Lambda F+\varepsilon\), where \(\mu\in\mathbb{R}^p\), \(\Lambda\in\mathbb{R}^{p\times q}\), \(F\sim N_q(0,I_q)\), and \(\varepsilon\sim N_p(0,\Psi)\). The factor vector \(F\) and specific-error vector \(\varepsilon\) are independent. The specific covariance matrix is diagonal, \(\Psi=\operatorname{diag}(\psi_1,\ldots,\psi_p)\), with \(\psi_i>0\) for each \(i\).

The zero mean and identity covariance for \(F\) set a conventional scale and origin for the factors. The vector \(\mu\) is the mean of the observed vector, since both \(F\) and \(\varepsilon\) have mean zero. The diagonal entries of \(\Psi\) are variances; the notation does not mean that the specific errors are absent. Independence of \(F\) and \(\varepsilon\) is an assumption of the model, not a conclusion from the covariance formula.

Deriving the Covariance Decomposition

Center the observed vector by subtracting its mean. The centered vector is \(\Lambda F+\varepsilon\). Its covariance has a contribution from the factors, a contribution from the specific errors, and two cross-covariance terms. Independence and zero means make the cross terms zero.

Theorem: Under the Gaussian common-factor model, \(\operatorname{E}[X]=\mu\) and \(\operatorname{Cov}(X)=\Sigma=\Lambda\Lambda^{\mathsf T}+\Psi\). Thus \(\Lambda\Lambda^{\mathsf T}\) is the factor-generated covariance component, and \(\Psi\) is the specific-error covariance component.

\begin{aligned} \operatorname{Cov}(X) &=\operatorname{E}\!\left[(X-\mu)(X-\mu)^{\mathsf T}\right]\\ &=\operatorname{E}\!\left[(\Lambda F+\varepsilon)(\Lambda F+\varepsilon)^{\mathsf T}\right]\\ &=\Lambda\operatorname{E}[FF^{\mathsf T}]\Lambda^{\mathsf T} +\Lambda\operatorname{E}[F\varepsilon^{\mathsf T}] +\operatorname{E}[\varepsilon F^{\mathsf T}]\Lambda^{\mathsf T} +\operatorname{E}[\varepsilon\varepsilon^{\mathsf T}]\\ &=\Lambda I_q\Lambda^{\mathsf T}+0+0+\Psi\\ &=\Lambda\Lambda^{\mathsf T}+\Psi . \end{aligned}

For the cross term, independence gives \(\operatorname{E}[F\varepsilon^{\mathsf T}] =\operatorname{E}[F]\operatorname{E}[\varepsilon]^{\mathsf T}=0\); the other cross term is its transpose. The mean follows directly from linearity of expectation: \(\operatorname{E}[X]=\mu+\Lambda\operatorname{E}[F]+\operatorname{E}[\varepsilon]=\mu\). These steps establish the decomposition without requiring any approximations.

Why the Observed Vector Is Gaussian

The Gaussian assumption applies to the latent factors and the specific errors. Independence then ensures that their sum after the linear transformation has a multivariate normal distribution. The following argument verifies this conclusion through all linear combinations.

Result: Under the model, \(X\sim N_p(\mu,\Sigma)\), where \(\Sigma=\Lambda\Lambda^{\mathsf T}+\Psi\). Since every \(\psi_i>0\), \(\Psi\) is positive definite, and therefore \(\Sigma\) is positive definite as well.

To prove the distributional statement, fix any \(a\in\mathbb{R}^p\). Then \(a^{\mathsf T}X=a^{\mathsf T}\mu+(\Lambda^{\mathsf T}a)^{\mathsf T}F+a^{\mathsf T}\varepsilon\). The last two terms are independent normal random variables because they are linear functions of independent Gaussian vectors. Their sum is normal. Its mean is zero, and its variance is \((\Lambda^{\mathsf T}a)^{\mathsf T}(\Lambda^{\mathsf T}a)+a^{\mathsf T}\Psi a =a^{\mathsf T}(\Lambda\Lambda^{\mathsf T}+\Psi)a\). Consequently every linear combination \(a^{\mathsf T}X\) is normal with the mean and variance specified by \(\mu\) and \(\Sigma\). This is the defining linear-combination characterization of a multivariate normal vector. Finally, for nonzero \(a\), \(a^{\mathsf T}\Sigma a=\|\Lambda^{\mathsf T}a\|^2+a^{\mathsf T}\Psi a>0\), because \(\Psi\) is positive definite. Thus \(\Sigma\) is positive definite. \(\square\)

The Gaussian assumption matters: the covariance identity itself follows from the stated means, covariances, and factor-error independence even without Gaussian distributions, but the conclusion \(X\sim N_p(\mu,\Sigma)\) does not. For example, a sum involving non-Gaussian factors need not be multivariate normal merely because its covariance has this form.

Worked Covariance Calculations

Worked Example: A One-Factor Model for Three Variables

Take \(p=3\), \(q=1\), and loading vector \(\Lambda=(0.8,0.6,0.5)^{\mathsf T}\). Let \(\Psi=\operatorname{diag}(0.36,0.64,0.75)\) and \(\mu=0\). Compute the observed covariance matrix.

With one factor, \(\Lambda\Lambda^{\mathsf T}\) is the outer product of the loading vector with itself:

\Lambda\Lambda^{\mathsf T} = \begin{pmatrix} 0.8\\0.6\\0.5 \end{pmatrix} \begin{pmatrix} 0.8&0.6&0.5 \end{pmatrix} = \begin{pmatrix} 0.64&0.48&0.40\\ 0.48&0.36&0.30\\ 0.40&0.30&0.25 \end{pmatrix}.

Add the specific covariance matrix to obtain

\Sigma= \begin{pmatrix} 0.64+0.36&0.48&0.40\\ 0.48&0.36+0.64&0.30\\ 0.40&0.30&0.25+0.75 \end{pmatrix} = \begin{pmatrix} 1&0.48&0.40\\ 0.48&1&0.30\\ 0.40&0.30&1 \end{pmatrix}.

For instance, the covariance between variables 1 and 2 is \(0.8(0.6)=0.48\), and the covariance between variables 2 and 3 is \(0.6(0.5)=0.30\). The diagonal entries each equal 1 because the corresponding factor contribution and specific variance sum to 1. The model therefore specifies \(X\sim N_3(0,\Sigma)\).

Worked Example: Two Factors and Three Variables

Let \(\Lambda=\begin{pmatrix}1&0\\0.5&1\\0&0.8\end{pmatrix}\) and \(\Psi=\operatorname{diag}(0.2,0.5,0.36)\). Find \(\Sigma\) and the covariance between variables 2 and 3.

The rows of \(\Lambda\) are \((1,0)\), \((0.5,1)\), and \((0,0.8)\). Their pairwise inner products give the entries of \(\Lambda\Lambda^{\mathsf T}\):

\Lambda\Lambda^{\mathsf T} = \begin{pmatrix} 1&0.5&0\\ 0.5&1.25&0.8\\ 0&0.8&0.64 \end{pmatrix}.

For example, the \((2,3)\) entry is \((0.5)(0)+(1)(0.8)=0.8\), while the \((2,2)\) entry is \((0.5)^2+1^2=1.25\). Adding \(\Psi\) changes the diagonal entries only:

\Sigma= \begin{pmatrix} 1.2&0.5&0\\ 0.5&1.75&0.8\\ 0&0.8&1 \end{pmatrix}.

The covariance between variables 2 and 3 is \(0.8\). In this model it comes from their shared-factor loading vectors: \((0.5,1)\) and \((0,0.8)\) have inner product \(0.8\). The specific covariance contributes nothing to this off-diagonal entry because \(\Psi\) is diagonal.

Worked Example: Distribution of a Linear Contrast

Suppose \(p=2\), \(q=1\), \(\mu=(2,-1)^{\mathsf T}\), \(\Lambda=(1,2)^{\mathsf T}\), and \(\Psi=\operatorname{diag}(1,3)\). Find the distribution of \(Y=X_1-X_2\).

First compute the observed covariance:

\Sigma= \begin{pmatrix}1\\2\end{pmatrix} \begin{pmatrix}1&2\end{pmatrix} + \begin{pmatrix}1&0\\0&3\end{pmatrix} = \begin{pmatrix}1&2\\2&4\end{pmatrix} + \begin{pmatrix}1&0\\0&3\end{pmatrix} = \begin{pmatrix}2&2\\2&7\end{pmatrix}.

Write \(Y=a^{\mathsf T}X\) with \(a=(1,-1)^{\mathsf T}\). Its mean is \(a^{\mathsf T}\mu=2-(-1)=3\). Its variance is

a^{\mathsf T}\Sigma a = \begin{pmatrix}1&-1\end{pmatrix} \begin{pmatrix}2&2\\2&7\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} =2+7-2(2)=5.

Thus \(Y\sim N(3,5)\). The same variance can be checked directly from the model: the factor coefficient in \(X_1-X_2\) is \(1-2=-1\), contributing variance \((-1)^2=1\); the difference of the two independent specific errors contributes \(1+3=4\). The total is \(1+4=5\).

Interpretation and a Common Pitfall

The decomposition is useful because it describes how the model attributes observed covariance to shared latent variation and specific variation. The loading matrix determines the shared covariance through its row-wise inner products: if rows \(i\) and \(j\) are denoted \(\lambda_i^{\mathsf T}\) and \(\lambda_j^{\mathsf T}\), their factor contribution is \(\lambda_i^{\mathsf T}\lambda_j\). The specific component is represented separately by \(\Psi\). Consequently, a proposed loading matrix and specific covariance matrix together imply a definite covariance matrix for the observed variables.

A common mistake is to say that the observed covariance is just \(\Lambda\Lambda^{\mathsf T}\), because the factors are the model’s main feature. That omits the variation in \(\varepsilon\). The correction is to include both terms, \(\Sigma=\Lambda\Lambda^{\mathsf T}+\Psi\); even with a zero-mean factor vector, the specific errors still contribute their covariance. A different mistake is to infer that any observed covariance matrix automatically has a unique factor-model interpretation. The decomposition states what covariance a chosen model implies; it does not, by itself, establish uniqueness of the model parameters.

Key takeaway: In the Gaussian common-factor model, \(X=\mu+\Lambda F+\varepsilon\) with independent standard Gaussian factors and Gaussian specific errors gives \(X\sim N_p(\mu,\Sigma)\), where \(\Sigma=\Lambda\Lambda^{\mathsf T}+\Psi\). Both covariance components are part of the model.

Check Your Understanding

Answer each question using the model assumptions and covariance decomposition.

What are the dimensions of \(\Lambda\) when there are \(p\) observed variables and \(q\) factors?
Why do the factor-error cross-covariance terms vanish in the derivation of \(\Sigma\)?
If two loading rows are \((1,2)\) and \((3,0)\), what is their shared-factor covariance contribution?
For a nonzero vector \(a\), explain why \(a^{\mathsf T}\Sigma a>0\) when every specific variance is positive.
Does the covariance identity alone imply that \(X\) is multivariate normal if the factors and errors are not Gaussian? Explain.