Assessing Fit Under Oblique and Orthogonal Solutions

Fit Is a Property of the Implied Covariance

Rotating a factor solution can change the apparent size and pattern of its loadings. That change does not necessarily mean that the model fits the data differently. The key distinction is between describing a loading matrix and assessing the covariance matrix implied by the model. Fit comparisons should be based on the latter.

Write the common-factor model for a \(p\)-dimensional observation \(X\) as \(X=\mu+\Lambda F+\varepsilon\), where \(\Lambda\) is a \(p\times m\) loading matrix, \(E[F]=0\), \(\operatorname{Cov}(F)=\Phi\), and \(\operatorname{Cov}(\varepsilon)=\Psi\). Assume \(\operatorname{Cov}(F,\varepsilon)=0\). Usually \(\Psi\) is diagonal, and in an oblique solution the factor variances are standardized so that \(\Phi\) has diagonal entries one. The model-implied covariance is

For an orthogonal solution, \(\Phi=I_m\). For an oblique solution, \(\Phi\) can have nonzero off-diagonal entries. Given a sample covariance matrix \(S\), the covariance residual matrix is \(R=S-\widehat{\Sigma}\), where \(\widehat{\Sigma}\) is the fitted model-implied covariance. Fit summaries based on \(R\), or on the likelihood of \(S\) under \(\widehat{\Sigma}\), evaluate the model's covariance reproduction. They are not summaries of the visual simplicity of \(\widehat{\Lambda}\).

Why Loadings Can Change While Covariance Fit Does Not

Suppose an orthogonal factor vector \(Z\) has covariance \(I_m\), and write \(X=\mu+\Lambda Z+\varepsilon\). Let \(A\) be an invertible \(m\times m\) matrix and define \(G=AZ\). Then \(\operatorname{Cov}(G)=AA^\mathsf{T}=\Phi\), and \(Z=A^{-1}G\). The same random vector \(X\) can therefore be written using the transformed loading matrix \(\Lambda_*=\Lambda A^{-1}\). Its common covariance is

The entries of \(\Lambda_*\) can differ substantially from those of \(\Lambda\), although the common covariance is identical. In particular, under an oblique solution the squared Euclidean length of a pattern-loading row is not generally the variable's communality. If \(\lambda_i^\mathsf{T}\) is row \(i\) of \(\Lambda\), its communality is \(\lambda_i^\mathsf{T}\Phi\lambda_i\); this reduces to the sum of squared loadings only when \(\Phi=I_m\). The structure matrix \(\Lambda\Phi\) and pattern matrix \(\Lambda\) also answer different questions when factors are correlated.

Residuals and Gaussian Likelihood

A residual matrix makes covariance discrepancies visible entry by entry. Its diagonal entries describe variance discrepancies; its off-diagonal entries describe discrepancies in covariances. Residual correlations can also be useful for diagnosing local misfit, but the residual matrix itself is scale-dependent. Any comparison requires the same observed covariance matrix and the same convention for centering and scaling.

For centered multivariate normal observations with sample covariance \(S\), suppose \(S\) and a candidate covariance matrix \(\Sigma\) are positive definite. The Gaussian maximum-likelihood discrepancy, apart from a common positive multiplier in the log-likelihood, is

For a sample of size \(n\), with the usual maximum-likelihood covariance convention, the difference between the maximized log-likelihood at \(\Sigma=S\) and the log-likelihood at \(\Sigma\) is \(nF_{\mathrm{ML}}/2\). Smaller discrepancy means better Gaussian covariance fit. It does not, by itself, establish that a rotated loading pattern is more interpretable.

Proof. If the implied covariance matrices are equal, then \(R=S-\widehat{\Sigma}\) is equal for both solutions. Their Gaussian densities depend on the factor representation only through the mean and covariance; with the same mean and \(\widehat{\Sigma}\), their likelihoods are equal.

For the inequality, define the positive-definite matrix \(B=\Sigma^{-1/2}S\Sigma^{-1/2}\), with eigenvalues \(b_1,\ldots,b_p>0\). The determinant identity gives \(|S|=|\Sigma||B|\), and cyclic invariance of trace gives \(\operatorname{tr}(S\Sigma^{-1})=\operatorname{tr}(B)\). Consequently,

For every \(b>0\), the function \(b-\log b-1\) has derivative \(1-1/b\), decreases up to \(b=1\), increases thereafter, and equals zero at \(b=1\). Thus equality requires every eigenvalue of \(B\) to equal one. Since \(B\) is symmetric positive definite, this means \(B=I_p\), or \(S=\Sigma\). \(\square\)

When Orthogonal and Oblique Solutions Can Differ in Fit

There are two important cases to distinguish. First, if both solutions are merely alternative representations of the same fitted covariance, their residuals and likelihoods must agree. In the unrestricted common-factor model, even the sets of possible common covariance matrices coincide for a fixed number of factors: any positive-semidefinite common covariance of rank at most \(m\) has a factorization \(CC^\mathsf{T}\) with at most \(m\) columns. It can therefore be represented with orthogonal factors. Allowing oblique coordinates does not automatically improve covariance fit in that setting.

Second, loading constraints can make the permitted covariance sets different. For example, a confirmatory pattern may require variable 1 to load only on factor 1 and variable 2 only on factor 2. With orthogonal factors this forces their common covariance to zero. With oblique factors it can be nonzero, because their covariance includes the factor correlation. When the same loading-zero constraints are imposed in both models and the oblique model permits \(\Phi=I_m\), the orthogonal model is a special case of the oblique model. Maximizing the same likelihood over the larger permitted set cannot give a lower maximum likelihood. That comparison concerns the constrained models, not the mere labels “orthogonal” and “oblique.” A likelihood difference alone also does not decide whether the added flexibility is warranted; parameterization and model-selection considerations matter.

Common Pitfall

A frequent error is to compare the sums of squared pattern loadings from an orthogonal and an oblique solution and conclude that the larger sum indicates better fit. That reasoning confuses a coordinate-dependent loading summary with a measure of covariance reproduction. Under oblique factors, even a row's squared pattern-loading length is not its communality; the correct expression is \(\lambda_i^\mathsf{T}\Phi\lambda_i\). To assess fit, compare the implied covariance matrices, their residuals, or their likelihoods on the same data. To compare interpretability, examine loading patterns and factor meaning separately.