Why Compare Factor-Extraction Methods?
Principal-axis, MINRES, and maximum-likelihood extraction all seek a common-factor representation of a covariance matrix, but they do not define “best fit” in the same way. Principal-axis extraction builds factors from a reduced covariance matrix; MINRES minimizes squared residual covariances off the diagonal; maximum likelihood optimizes a likelihood-based discrepancy. When the model fits a covariance matrix exactly, these approaches can agree. With imperfect sample data, their objectives can lead to different loadings and uniqueness estimates.
The comparison below uses one four-variable covariance matrix with a known one-factor representation. This gives a transparent benchmark: we can calculate the loadings, uniquenesses, and fitted covariances, then check exactly what each method is optimizing. Agreement in this example is a consequence of exact fit, not a claim that the methods generally produce identical estimates.
For one factor, write the loading vector as \(\lambda=(\lambda_1,\ldots,\lambda_p)^{\mathsf T}\). Then each fitted covariance between distinct variables \(i\) and \(j\) is \(\lambda_i\lambda_j\), and the fitted variance of variable \(i\) is \(\lambda_i^2+\psi_i\), where \(\psi_i\) is its uniqueness. In a standardized-variable example, the observed variances are all one, so a fitted solution that reproduces those variances has \(\psi_i=1-\lambda_i^2\).
A Covariance Matrix with an Exact One-Factor Representation
Consider the covariance matrix below. Its diagonal entries are one, so it is also a correlation matrix. The off-diagonal entries are deliberately chosen to have a simple common-factor representation.
Take \(\lambda=(0.8,0.6,0.5,0.4)^{\mathsf T}\) and \(\Psi=\operatorname{diag}(0.36,0.64,0.75,0.84)\). The common covariance between variables 1 and 2, for instance, is \(0.8(0.6)=0.48\). The other five products are \(0.8(0.5)=0.40\), \(0.8(0.4)=0.32\), \(0.6(0.5)=0.30\), \(0.6(0.4)=0.24\), and \(0.5(0.4)=0.20\). The diagonal entries are also one: \(0.8^2+0.36=1\), \(0.6^2+0.64=1\), \(0.5^2+0.75=1\), and \(0.4^2+0.84=1\). Thus \(S=\lambda\lambda^{\mathsf T}+\Psi\).
Worked Example: Verify the Covariance Model and Its Uniquenesses
The loadings are \(0.8, 0.6, 0.5, 0.4\). For standardized variables, the communality is the variance attributed to the common factor, \(\lambda_i^2\), and the uniqueness is the remaining variance, \(1-\lambda_i^2\).
For variable 1, the communality is \(0.8^2=0.64\), so its uniqueness is \(1-0.64=0.36\). For variable 2, these values are \(0.6^2=0.36\) and \(1-0.36=0.64\). For variables 3 and 4 they are \(0.5^2=0.25\) with uniqueness \(0.75\), and \(0.4^2=0.16\) with uniqueness \(0.84\).
The model therefore reconstructs every diagonal entry as common variance plus uniqueness, and every off-diagonal entry as a product of loadings. All uniquenesses are strictly positive, so this representation does not rely on a boundary estimate.
Principal-Axis Extraction
Principal-axis extraction uses a reduced covariance matrix: it retains the observed off-diagonal covariances but replaces each diagonal variance by an estimated communality. For a one-factor solution with communalities \(h_i^2\), this reduced matrix is \(S^*=S-\operatorname{diag}(\psi_1,\ldots,\psi_p)\), where \(\psi_i=S_{ii}-h_i^2\). An iteration updates the communalities using the extracted factor solution. The following calculation identifies an exact fixed point for the matrix in this example.
Worked Example: Calculate the Principal-Axis Solution
Use the communalities \(h^2=(0.64,0.36,0.25,0.16)\), the squares of the proposed loadings. Subtract the corresponding uniquenesses from the diagonal of \(S\). Since each observed variance is one, the resulting reduced matrix is
The matrix is rank one. Its nonzero eigenvalue is \(\lambda^{\mathsf T}\lambda=0.64+0.36+0.25+0.16=1.41\); a corresponding unit eigenvector is \(\lambda/\sqrt{1.41}\). Principal-axis extraction recovers the loading vector by multiplying this eigenvector by the square root of the eigenvalue:
The extracted loadings therefore reproduce the communalities used to form \(S^*\). They are a fixed point of the communality update, with uniquenesses \((0.36,0.64,0.75,0.84)\). The overall sign of the one-factor loading vector could be reversed without changing any fitted covariance.
This derivation shows why principal-axis extraction recovers the exact representation here: once the diagonal communalities are correct, the reduced matrix is precisely the common covariance matrix, and its leading eigenpair returns the factor loadings. In applications the communalities are not known in advance, so the method must estimate and update them. The calculation above verifies the fixed point; it does not imply that every possible initialization or dataset leads to this solution.
MINRES and the Off-Diagonal Residuals
MINRES, short for minimum residual, judges a one-factor solution by the discrepancies between observed and fitted off-diagonal covariances. A common form of its objective is
The sum omits the diagonal: the method targets the pattern of covariances between distinct variables, rather than adding squared variance residuals to this objective. A reported solution may still give uniqueness estimates to account for the fitted variances.
Worked Example: Check the MINRES Residual Objective
For the loading vector \((0.8,0.6,0.5,0.4)^{\mathsf T}\), the six fitted off-diagonal entries are \(0.48, 0.40, 0.32, 0.30, 0.24,\) and \(0.20\), in the same order as the corresponding entries of \(S\). Each residual is observed minus fitted, so all six residuals equal zero.
Consequently,
A sum of squares cannot be negative, so zero is the global minimum possible value of this objective. The fitted diagonal is also correct when the uniquenesses are \((0.36,0.64,0.75,0.84)\), although those diagonal matches are not additional terms in the stated MINRES objective.
The zero-residual result depends on the exact fit. To see how the objective responds to a different candidate, take loadings \((0.75,0.65,0.55,0.45)^{\mathsf T}\). Their fitted off-diagonal covariances are \(0.4875, 0.4125, 0.3375, 0.3575, 0.2925,\) and \(0.2475\). Subtracting these from the entries of \(S\) gives residuals \(-0.0075,-0.0125,-0.0175,-0.0575,-0.0525,\) and \(-0.0475\). Thus the objective at this candidate is
This is an objective check, not a claim that the alternative loading vector is a MINRES solution. It illustrates that MINRES compares the off-diagonal residual pattern directly.
Maximum Likelihood and the Covariance Discrepancy
For Gaussian factor analysis, maximum-likelihood extraction chooses model parameters to maximize the Gaussian likelihood. When the sample covariance \(S\) and fitted covariance \(\widehat{\Sigma}\) are positive definite, the corresponding likelihood discrepancy can be written as
The ML solution minimizes this discrepancy over covariance matrices allowed by the factor model. Unlike the MINRES objective, this expression uses the full covariance matrices, including their diagonals, and incorporates their determinants and inverse.
Proof. Let \(A=\widehat{\Sigma}^{-1/2}S\widehat{\Sigma}^{-1/2}\), and let its positive eigenvalues be \(x_1,\ldots,x_p\). The trace satisfies \(\operatorname{tr}(S\widehat{\Sigma}^{-1})=\operatorname{tr}(A)=\sum_i x_i\). Also, \(|S|=|\widehat{\Sigma}|\,|A|\), so \(\log|\widehat{\Sigma}|-\log|S|=-\log|A|=-\sum_i\log x_i\). Therefore,
For every \(x>0\), \(x-\log x-1\geq 0\): its derivative is \(1-1/x\), so it decreases up to \(x=1\), increases after \(x=1\), and equals zero at \(x=1\). Hence the sum is nonnegative. Equality requires every \(x_i=1\), which means \(A=I\), or equivalently \(S=\widehat{\Sigma}\). \(\square\)
Worked Example: Evaluate the ML Discrepancy at the Fitted Covariance
For the factor solution above, \(\widehat{\Sigma}=S\). The determinant terms in \(F_{\text{ML}}\) cancel, and \(S\widehat{\Sigma}^{-1}=I_4\), whose trace is 4. Since \(p=4\),
The covariance matrix is positive definite because it is the sum of a positive semidefinite matrix \(\lambda\lambda^{\mathsf T}\) and the positive definite diagonal matrix \(\Psi\), whose diagonal entries are all positive. The theorem therefore applies. The model reaches the smallest possible ML discrepancy, and the uniquenesses and loadings are the same as those found in the exact principal-axis and MINRES calculations.
What Agreement Does—and Does Not—Tell Us
For this matrix, all three methods recover the same fitted covariance matrix and the same one-factor representation, up to reversing the factor’s sign. The reason is stronger than mere numerical similarity: every observed covariance is reproduced exactly. Principal-axis extraction has a reduced-matrix fixed point, the MINRES residual objective reaches its lower bound of zero, and the ML discrepancy reaches its lower bound of zero.
With an imperfect sample covariance matrix, a one-factor model will generally not reproduce every entry. The methods can then make different compromises. MINRES minimizes squared off-diagonal residuals; ML minimizes its determinant-and-trace discrepancy under the Gaussian model; principal-axis extraction works through communalities and a reduced matrix. The resulting loading estimates and uniquenesses need not coincide. Thus, when comparing solutions, inspect the fitted covariance matrices and uniqueness estimates as well as the loading vectors, and identify the objective each solution optimizes.
Check Your Understanding
Use the definitions and calculations above to check your understanding of how the three extraction methods compare.
- For the four-variable example, calculate the fitted covariance between variables 2 and 4 from their loadings.
- Why does the reduced matrix in the principal-axis calculation have rank one?
- Which entries contribute to the MINRES objective as defined here: diagonal residuals, off-diagonal residuals, or both?
- What condition makes the ML discrepancy equal zero when both covariance matrices are positive definite?
- Explain why identical solutions in this example do not imply that the three methods will give identical estimates for arbitrary sample covariance matrices.