Common-Factor Analysis Versus Principal Components

Two Methods, Two Covariance Targets

Common-factor analysis and principal components are often displayed as similar-looking loading matrices. Both can reduce a large collection of observed variables to a smaller set of dimensions. But their mathematical targets differ: common-factor analysis models shared covariance, while principal components summarize variation in the observed variables. The distinction matters when interpreting a loading, a variance percentage, or the number of dimensions to retain.

Write \(X\) for a \(p\)-dimensional random vector of centered observed variables and \(\Sigma=\operatorname{Cov}(X)\) for its covariance matrix. In applications, \(\Sigma\) is often estimated by a sample covariance matrix \(S\). The population formulas below describe the targets; replacing \(\Sigma\) by \(S\) gives their sample counterparts.

Definition: In the common-factor model, \(X=\Lambda F+\varepsilon\), where \(F\) is a vector of \(q\) common factors, \(\Lambda\) is the \(p\times q\) loading matrix, and \(\varepsilon\) is a vector of specific or error terms. Assume \(E[F]=0\), \(E[\varepsilon]=0\), \(\operatorname{Cov}(F)=\Phi\), \(\operatorname{Cov}(\varepsilon)=\Psi\), and \(\operatorname{Cov}(F,\varepsilon)=0\). The usual model also assumes \(\Psi\) is diagonal. Then the model-implied covariance is \(\Sigma=\Lambda\Phi\Lambda'+\Psi\). The matrix \(\Lambda\Phi\Lambda'\) is the common covariance; \(\Psi\) contains the variable-specific variances.

The factors are not observed variables being selected to capture the largest possible spread. Instead, the factor model proposes that correlations among observed variables arise, in part, because they share latent sources of variation. Its parameters are chosen to make the model-implied covariance agree with the covariance being analyzed, according to the extraction or estimation criterion. Different criteria, such as maximum likelihood and least squares, define different ways to measure that agreement.

Definition: For principal components, let \(v_1,\ldots,v_p\) be orthonormal eigenvectors of \(\Sigma\), ordered so their eigenvalues satisfy \(\lambda_1\ge\cdots\ge\lambda_p\ge0\). The \(j\)th principal component is the linear combination \(v_j'X\), with variance \(\lambda_j\). The first \(k\) components are chosen to maximize the total variance of \(k\) orthonormal linear combinations of the observed variables.

How the Variance Accounting Differs

The factor model assigns each variable’s variance to a common part and a unique part. Under the assumptions in the definition, \(\operatorname{Cov}(X)=\operatorname{Cov}(\Lambda F+\varepsilon)\). Expanding this covariance gives \(\Lambda\operatorname{Cov}(F)\Lambda'+\operatorname{Cov}(\varepsilon)\), because the cross-covariance terms vanish. Thus the model implies \(\Sigma=\Lambda\Phi\Lambda'+\Psi\). For variable \(i\), its communality is \(h_i^2=(\Lambda\Phi\Lambda')_{ii}\), and its uniqueness is \(\psi_i=\Psi_{ii}\). Therefore \(\operatorname{Var}(X_i)=h_i^2+\psi_i\).

For principal components, the eigenvalues partition total observed variance: \(\operatorname{tr}(\Sigma)=\sum_{j=1}^{p}\lambda_j\). The first \(k\) components account for \(\sum_{j=1}^{k}\lambda_j\), so their proportion of total variance is \(\sum_{j=1}^{k}\lambda_j/\operatorname{tr}(\Sigma)\), provided \(\operatorname{tr}(\Sigma)>0\). This accounting does not separate common variance from uniqueness. A component can capture variation due to both.

For oblique factors, \(\Phi\) need not be the identity matrix. The communality remains the diagonal element of \(\Lambda\Phi\Lambda'\), but it need not be interpreted as a sum of independent, separately attributable contributions from each factor: correlated factors can contribute cross-terms. In contrast, the orthogonal principal components have uncorrelated scores, and their variances add directly to total variance.

Worked Example: A One-Factor Model and the First Principal Component

Suppose three centered variables follow a one-factor model with loading vector \(\Lambda=(1,1,1)'\), factor variance \(\Phi=1\), and \(\Psi=I_3\). The implied covariance is \(\Sigma=\mathbf{1}\mathbf{1}'+I_3\), where \(\mathbf{1}=(1,1,1)'\). Written out,

\Sigma= \begin{pmatrix} 2&1&1\\ 1&2&1\\ 1&1&2 \end{pmatrix}.

The common covariance is \(\mathbf{1}\mathbf{1}'\), whose diagonal entries are all 1. Each variable has variance 2, comprising communality 1 and uniqueness 1. Across the three variables, the common part contributes \(1+1+1=3\) to the trace, and the unique part contributes another 3. Total variance is 6.

The unit vector \(v_1=\mathbf{1}/\sqrt{3}\) is an eigenvector of \(\Sigma\): since \(\mathbf{1}'\mathbf{1}=3\), \(\Sigma\mathbf{1}=4\mathbf{1}\). Its eigenvalue is 4. Any vector orthogonal to \(\mathbf{1}\) has eigenvalue 1, because \(\mathbf{1}'v=0\) implies \(\Sigma v=v\). Thus the first component explains \(4/6=2/3\) of total variance.

The first component’s variance is 4, not 3. The additional 1 is the variance contributed by the specific terms when projected onto \(v_1\): \(v_1'I_3v_1=1\). PCA has captured variation in the observed variables, including some uniqueness; the factor model’s common variance and PCA’s component variance are different quantities.

Why Principal Components Maximize Variance

The optimization goal for PCA can be stated without reference to a factor model. A \(k\)-dimensional set of orthonormal directions can be represented by a \(p\times k\) matrix \(U\) satisfying \(U'U=I_k\). The variance summed across the corresponding component scores is \(\operatorname{tr}(U'\Sigma U)\).

Theorem: If \(\Sigma\) is a symmetric positive semidefinite \(p\times p\) matrix with eigenvalues \(\lambda_1\ge\cdots\ge\lambda_p\), then \(\operatorname{tr}(U'\Sigma U)\le\sum_{j=1}^{k}\lambda_j\) for every \(p\times k\) matrix \(U\) with \(U'U=I_k\). Equality is attained by taking the columns of \(U\) to be eigenvectors for the \(k\) largest eigenvalues. If eigenvalues tie at the boundary, the maximizing subspace need not be unique.

Proof. Let \(v_1,\ldots,v_p\) be an orthonormal eigenbasis of \(\Sigma\), and let \(u_1,\ldots,u_k\) be the columns of \(U\). Define \(a_j=\sum_{r=1}^{k}(v_j'u_r)^2\). Each \(a_j\) is between 0 and 1: it is the squared length of the projection of the unit vector \(v_j\) onto the column space of \(U\). Also, \(\sum_{j=1}^{p}a_j=k\), since \(\sum_j a_j=\sum_{r=1}^k\sum_j(v_j'u_r)^2=\sum_{r=1}^k u_r'u_r=k\).

Expanding each \(u_r\) in the eigenbasis gives \(\operatorname{tr}(U'\Sigma U)=\sum_{j=1}^{p}\lambda_j a_j\). Since the eigenvalues are ordered, the largest possible weighted sum subject to \(0\le a_j\le1\) and \(\sum_j a_j=k\) puts weight 1 on the first \(k\) eigenvalues and weight 0 on the rest. To see the bound directly, let \(c=\lambda_k\). Then

\sum_{j=1}^{p}\lambda_j a_j \le \sum_{j=1}^{k}\lambda_j a_j +c\sum_{j=k+1}^{p}a_j =\sum_{j=1}^{k}\bigl[\lambda_j a_j+c(1-a_j)\bigr] \le\sum_{j=1}^{k}\lambda_j.

The first inequality uses \(\lambda_j\le c\) for \(j>k\); the equality uses \(\sum_{j>k}a_j=\sum_{j\le k}(1-a_j)\). The last inequality uses \(\lambda_j\ge c\) for \(j\le k\). Choosing the first \(k\) eigenvectors achieves the bound, proving the result. \(\square\)

This theorem explains PCA’s criterion: it selects a subspace that maximizes projected observed variance. It does not say that the subspace is the best-fitting latent-factor model. A common-factor analysis instead seeks parameters for which \(\Lambda\Phi\Lambda'+\Psi\) accounts for the covariance under its specified model and fitting criterion.

Worked Example: Unequal Uniquenesses Change the Direction

Let one common factor have loading vector \(\Lambda=(1,1,1)'\), factor variance 1, and uniqueness matrix \(\Psi=\operatorname{diag}(1,2,3)\). The implied covariance is

\Sigma= \begin{pmatrix} 2&1&1\\ 1&3&1\\ 1&1&4 \end{pmatrix}.

Each variable has communality 1. Their variances are 2, 3, and 4, respectively, so the total variance is 9; the common covariance has trace 3 and the uniqueness matrix has trace 6.

The unit direction of the common loading vector is \(u=(1,1,1)'/\sqrt{3}\). Its variance is \(u'\Sigma u=15/3=5\), because \(\Sigma(1,1,1)'=(4,5,6)'\). But \(u\) is not an eigenvector: \((4,5,6)'\) is not proportional to \((1,1,1)'\). Therefore it is not a principal-component direction.

Indeed, the unit vector in the direction \(x=(4,5,6)'\) has variance \(x'\Sigma x/(x'x)\). Here \(x'x=16+25+36=77\), and \(\Sigma x=(19,25,33)'\), so \(x'\Sigma x=4(19)+5(25)+6(33)=399\). Its variance is \(399/77\approx5.18\), which exceeds 5. The common-factor loading direction captures the shared factor, but PCA can choose a different direction that yields greater total observed variance.

Covariance Versus Correlation and the Role of Scale

PCA based on \(\Sigma\) is sensitive to measurement units: a variable with a larger numerical scale can contribute more to the total variance. PCA based on the correlation matrix instead analyzes standardized variables, giving each variable unit variance before the decomposition. Neither choice is universally correct; it depends on whether the original scales or equalized scales answer the scientific question.

Factor analysis can also be fitted to a covariance matrix or a correlation matrix. The choice changes the quantities being modeled, but does not make factor analysis identical to PCA. In either case, the factor model distinguishes common covariance from uniqueness; PCA’s eigenvalue decomposition partitions the total variance of the matrix supplied to it.

Worked Example: Rescaling Changes Covariance-Based PCA

Suppose two standardized variables have correlation matrix \(R=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). Its first eigenvector is \((1,1)'/\sqrt{2}\), with eigenvalue 1.8; the other eigenvalue is 0.2. PCA on the correlation matrix therefore selects the equal-weight direction.

Now measure the first variable in units ten times as large, leaving the second unchanged. The new covariance matrix is \(DRD\), where \(D=\operatorname{diag}(10,1)\):

DRD= \begin{pmatrix} 100&8\\ 8&1 \end{pmatrix}.

Its eigenvalues solve \(\lambda^2-101\lambda+36=0\), so the larger is \((101+\sqrt{10057})/2\approx100.64\). For an eigenvector \((1,r)'\) at that eigenvalue, the second row gives \(8+r=\lambda r\), hence \(r=8/(\lambda-1)\approx0.0803\). After normalization, the first principal direction is approximately \((0.9968,0.0800)'\), much closer to the first variable’s axis than the equal-weight direction. The change is a consequence of scaling, not a change in the underlying correlation.

Common Mistake: Treating PCA as a Factor Model

A frequent mistake is to say that the first few principal components are “the common factors” because they explain a large proportion of variance. That conclusion does not follow. PCA maximizes variance in linear combinations of the observed variables, and its variance percentages include any observed variation, including variable-specific variation. Factor analysis posits a particular covariance structure that separates common covariance and uniqueness. A PCA solution may be useful for dimension reduction, but interpreting its components as latent causes requires assumptions beyond PCA’s optimization.

A related pitfall is to compare a factor’s common variance directly with a PCA eigenvalue. In the three-variable example above, the common covariance trace is 3, while the leading eigenvalue is 4. Those numbers answer different questions: the former measures modeled shared variance across variables, and the latter measures variance along one unit direction in the observed space.

Key takeaway: Principal components are defined by a variance-maximization problem for observed variables. Common factors are defined through a covariance model that separates shared covariance from uniqueness. Their loadings, variance decompositions, and interpretations are therefore not interchangeable.

Check Your Understanding

Use the definitions and distinctions above to answer these questions.

In a common-factor model, what assumptions make the covariance equal to \(\Lambda\Phi\Lambda'+\Psi\)?
Why can a principal component capture variance that the factor model classifies as unique?
For the covariance matrix in the unequal-uniqueness example, why is the normalized loading vector not a principal-component direction?
What quantity does the ratio of the first \(k\) eigenvalues’ sum to \(\operatorname{tr}(\Sigma)\) describe?
How does rescaling one variable affect covariance-based PCA, and why might correlation-based PCA respond differently?