Worked Diagnostic Comparison of Competing Rotated Models

Why Compare Rotated Solutions Systematically?

A rotated loading matrix can tell a compelling substantive story and still leave important covariance unexplained. Another solution may fit the observed covariance better but have less distinct loading patterns or factors that are harder to describe. A useful comparison therefore examines several kinds of evidence together: whether factors match across solutions, how concentrated the loadings are, whether factor correlations are plausible, where residuals remain, and how fit criteria balance fit with complexity.

These diagnostics answer different questions. Congruence describes similarity after factors are aligned; it does not establish that two models fit equally well. A simple loading pattern can aid interpretation, but simplicity alone does not establish adequate fit. Information criteria summarize a fit-complexity tradeoff, but do not identify which factor interpretation is scientifically useful.

Definition: For a fitted factor solution with loading matrix \(\Lambda\), factor correlation matrix \(\Phi\), and diagonal uniqueness matrix \(\Psi\), the model-implied covariance matrix is \(\Sigma=\Lambda\Phi\Lambda^\mathsf{T}+\Psi\). A diagnostic comparison evaluates the candidate solutions using both their loading representations and their fit to the same observed data. For standardized variables, the corresponding comparison can be made on the correlation scale.

Align the Factors Before Comparing Loadings

A factor may appear in a different column, or with every loading sign reversed, in a second solution. Those changes do not by themselves indicate a different substantive factor. Before comparing corresponding columns, match factors by their loading patterns and allow a sign reversal when it improves alignment. Tucker’s congruence coefficient summarizes the similarity of two aligned loading columns:

\phi_j=\frac{\boldsymbol{\lambda}_{Aj}^{\mathsf{T}}\boldsymbol{\lambda}_{Bj}} {\sqrt{\left(\boldsymbol{\lambda}_{Aj}^{\mathsf{T}}\boldsymbol{\lambda}_{Aj}\right) \left(\boldsymbol{\lambda}_{Bj}^{\mathsf{T}}\boldsymbol{\lambda}_{Bj}\right)}}.

Here \(\boldsymbol{\lambda}_{Aj}\) and \(\boldsymbol{\lambda}_{Bj}\) are the loading columns judged to represent the same factor. The coefficient is a cosine similarity: its value is near 1 when the columns point in similar directions, near \(-1\) when their signs are reversed, and near 0 when they are nearly orthogonal. It is descriptive, not a test of equality, and its interpretation depends on the variables being the same and on a sensible factor matching.

Worked Example: Matching Factors and Comparing Simplicity

Suppose two solutions for six standardized variables have these loading columns, already matched by their high-loading variables:

\boldsymbol{\lambda}_{A1}=(.80,.75,.10,.15,.55,.45)^\mathsf{T},\quad \boldsymbol{\lambda}_{B1}=(.78,.72,.12,.18,.62,.30)^\mathsf{T}.

For the first factor, the dot product is \[ .80(.78)+.75(.72)+.10(.12)+.15(.18)+.55(.62)+.45(.30)=1.679. \] The squared lengths are \(1.740\) and \(1.648\), respectively. Therefore, \[ \phi_1=\frac{1.679}{\sqrt{1.740(1.648)}}\approx\frac{1.679}{1.6934}=.992. \] The first loading pattern is highly similar across the solutions.

For the second factor, take \(\boldsymbol{\lambda}_{A2}=(.10,.15,.78,.72,.45,.50)^\mathsf{T}\) and \(\boldsymbol{\lambda}_{B2}=(.12,.18,.76,.70,.30,.62)^\mathsf{T}\). Their dot product is \(1.5808\); their squared lengths are \(1.6118\) and \(1.5888\). Thus \[ \phi_2=\frac{1.5808}{\sqrt{1.6118(1.5888)}}\approx .988. \] Both factors have closely aligned loading patterns, though this says nothing by itself about which solution fits better.

A simple descriptive loading-complexity summary can also be calculated if the intended primary factor for each variable is specified in advance. Assign variables 1 and 2 to factor 1, variables 3 and 4 to factor 2, variable 5 to factor 1, and variable 6 to factor 2. Sum the absolute nonprimary loadings. For solution A the sum is \(.10+.15+.10+.15+.45+.45=1.40\); for solution B it is \(.12+.18+.12+.18+.30+.30=1.20\). Under this particular summary, B has smaller cross-loadings. The result is descriptive: a different substantive assignment or a different simplicity measure could lead to a different comparison.

Factor Correlations and Residual Diagnostics

For an oblique solution, \(\Phi\) records correlations among the factors; for an orthogonal solution, \(\Phi\) is the identity matrix. Factor correlations help distinguish a solution with clearly separated dimensions from one in which the estimated factors substantially overlap. They should be interpreted in context: a moderate correlation may be scientifically plausible, while a very high correlation can make distinct factor labels difficult to defend. It is not enough to prefer the solution with the smallest correlations automatically.

Residuals show how the fitted covariance or correlation matrix differs from the observed one. On a correlation scale, define \(e_{ij}=r_{ij}-\hat r_{ij}\) for each off-diagonal pair, where \(r_{ij}\) is the observed correlation and \(\hat r_{ij}\) is its model-implied value. One useful summary is the root mean squared residual (RMSR):

\operatorname{RMSR}=\sqrt{\frac{1}{m}\sum_{i<j}e_{ij}^{\,2}}, \qquad m=\frac{p(p-1)}{2}.

The summary measures average residual size, while individual residuals reveal where misfit is concentrated. In particular, a modest RMSR can coexist with one or more notable residuals. Residual diagnostics are therefore best reported with both an overall summary and a description of the largest or substantively important residuals.

Worked Example: Comparing Residual Patterns

For six variables there are \(m=6(5)/2=15\) off-diagonal residual correlations. Suppose the 15 residuals for solution A are \[ .02,-.03,.01,.04,-.02,\ .01,.02,-.04,.03,.02,\ -.01,.05,-.02,.01,.03. \] Their squared values sum to \(0.0108\). Hence \[ \operatorname{RMSR}_A=\sqrt{\frac{0.0108}{15}} =\sqrt{0.00072}=0.0268. \] The largest absolute residual is \(0.05\).

For solution B, suppose the residuals are \[ .03,-.02,.02,.06,-.01,\ .02,.03,-.05,.04,.03,\ -.02,.07,-.03,.02,.04. \] Their squared values sum to \(0.0199\), so \[ \operatorname{RMSR}_B=\sqrt{\frac{0.0199}{15}} =\sqrt{0.0013267}=0.0364. \] The largest absolute residual is \(0.07\). Both the average residual size and the largest residual are greater for B in this example. That is evidence favoring A on these diagnostics, but the residual locations and their substantive importance still matter.

Fit Criteria and What They Can Establish

When two fitted models are estimated from the same observations using comparable likelihoods, Akaike’s information criterion (AIC) and the Bayesian information criterion (BIC) summarize fit while penalizing the number of estimated parameters. With maximized log-likelihood \(\ell\), sample size \(n\), and \(k\) free parameters, use

\operatorname{AIC}=-2\ell+k\text{ penalty }= -2\ell+2k, \qquad \operatorname{BIC}=-2\ell+k\log(n).

Lower values are preferred within the set of models being compared, but the criteria do not identify the best substantive interpretation. AIC and BIC are meaningful comparisons only when the data and likelihood basis are the same and parameter counts are handled consistently. The parameter count should reflect the fitted model after its identification constraints; it should not count arbitrary choices of factor signs or column order as extra parameters.

Worked Example: Combining Fit and Complexity

Suppose \(n=300\) observations on six variables are fitted using two candidate models. Model A is an oblique model with \(k_A=30\) free parameters and \(-2\ell_A=4010\). Model B is an orthogonal model with \(k_B=29\) and \(-2\ell_B=4022\). These are fitted-model outputs, and both likelihoods use the same observations and comparable likelihood definitions.

For A, \[ \operatorname{AIC}_A=4010+2(30)=4070,\qquad \operatorname{BIC}_A=4010+30\log(300)\approx4010+171.11=4181.11. \] For B, \[ \operatorname{AIC}_B=4022+2(29)=4080,\qquad \operatorname{BIC}_B=4022+29\log(300)\approx4022+165.41=4187.41. \] Both criteria favor A, despite its one additional parameter. The improvement in \(-2\ell\), from 4022 to 4010, is large enough to outweigh the extra penalty in these calculations. This is evidence for A under these criteria, not proof that its factor interpretation is correct.

Why Equivalent Rotations Must Have the Same Fit

A crucial check is whether two outputs are genuinely competing fitted models or merely different coordinate descriptions of the same fitted covariance structure. For a nonsingular change of factor coordinates \(T\), define \(\Lambda^*=\Lambda T\) and \(\Phi^*=T^{-1}\Phi T^{-\mathsf{T}}\). Then

\Lambda^*\Phi^*(\Lambda^*)^\mathsf{T} =\Lambda T(T^{-1}\Phi T^{-\mathsf{T}})T^\mathsf{T}\Lambda^\mathsf{T} =\Lambda\Phi\Lambda^\mathsf{T}.

The uniqueness matrix is unchanged, so the implied covariance is unchanged as well. This is the covariance-preserving property of an equivalent factor rotation.

Result: If two solutions are equivalent rotations of the same fitted model, use the same observations, and have the same effective parameter count, they have the same likelihood, AIC, and BIC. Differences in these quantities indicate that something beyond a covariance-preserving rotation differs, such as a constraint, a fitted covariance structure, or numerical convergence.

To see the information-criterion implication, identical implied covariance matrices give identical likelihoods for the same likelihood model and data. If also \(k_A=k_B=k\), then \[ \operatorname{AIC}_A-\operatorname{AIC}_B=(-2\ell_A+2k)-(-2\ell_B+2k)=0, \] and \[ \operatorname{BIC}_A-\operatorname{BIC}_B=(-2\ell_A+k\log n)-(-2\ell_B+k\log n)=0. \] Thus, a claim that one pure rotation has better AIC or BIC requires checking whether the fitted models or parameter counts actually differ.

Make the Reporting Choice Explicit

A transparent comparison does not hide disagreement among diagnostics. State which evidence favors each candidate, indicate whether the candidates are equivalent or constrained differently, and explain the substantive criterion used to make the final choice. A compact comparison table can make that reasoning auditable.

Diagnostic	Question it addresses	How to use it
Congruence	Do aligned loading patterns resemble one another?	Match columns and signs first; treat the result as descriptive similarity.
Loading simplicity	Are loadings concentrated on intended factors?	Specify the primary-factor assignment or summary before comparing.
Factor correlations	Are the dimensions distinct enough for the proposed interpretation?	Compare with substantive expectations; do not minimize mechanically.
Residuals	Which observed covariances remain unexplained?	Report an overall summary and inspect the largest residuals.
AIC and BIC	Does the likelihood improvement justify model complexity?	Compare only on the same data and a consistent likelihood and parameter-count basis.

For example, if an orthogonal solution has simpler loadings but leaves larger residual correlations, while an oblique solution fits better and has a plausible factor correlation, reporting the oblique solution may be defensible. The report should say why that tradeoff is acceptable and note the alternative. If the factor correlation is implausibly high or the extra fit is driven by a few local residuals, the same numbers might support a more cautious interpretation.

Common Mistake: Treating One Diagnostic as a Verdict

Wrong reasoning: “The loading congruence is .99, so the solutions are interchangeable and the one with simpler loadings must be the better model.” Congruence measures similarity of matched loading columns, not equality of residuals, implied covariances, factor correlations, or likelihoods. A simple pattern is not sufficient evidence of adequate fit.

Correction: First determine whether the solutions are covariance-equivalent rotations or competing constrained fits. Then compare aligned loading patterns, factor correlations, residual summaries and locations, and fit criteria on a consistent basis. State which diagnostic drives the reporting choice and what uncertainty or tradeoff remains. This keeps interpretability and statistical fit visible rather than collapsing them into a single score.

Key takeaway: A defensible choice among rotated solutions is a reasoned synthesis, not a contest with one universal winning statistic. Align factors, assess simplicity and factor correlations, inspect residuals, compare fit criteria appropriately, and report the tradeoffs.

Check Your Understanding

Use the definitions and comparison logic above to answer these questions.

Why must factor columns be matched, including possible sign reversals, before interpreting a congruence coefficient?
What does a smaller sum of absolute cross-loadings establish, and what does it not establish?
Why should residual RMSR be reported alongside inspection of individual residuals?
Two solutions are pure covariance-preserving rotations with the same parameter count and likelihood basis. What should happen to their AIC and BIC, and why?
In the worked fit example, which criterion values favor model A, and how does the additional parameter affect the comparison?
Give one reason a solution with simpler loadings might still be a poor reporting choice.