Why Factor Loadings Are Not Unique
A fitted factor model summarizes dependence among measured variables using a smaller set of latent factors. But the factor axes are not automatically determined by the fitted covariance: different loading matrices can encode the same model. This matters when interpreting genomic profiles, because a loading can look biologically specific or diffuse depending on the chosen factor coordinates, even when the model-implied covariance has not changed.
Use the factor model \(x=\mu+\Lambda f+\varepsilon\), where \(x\) is a \(p\)-vector of measured variables, \(\mu\) is its mean vector, \(\Lambda\) is a \(p\times k\) loading matrix, \(f\) is a \(k\)-vector of factors, and \(\varepsilon\) is a \(p\)-vector of residuals. Assume \(E(f)=0\), \(E(\varepsilon)=0\), \(\operatorname{Cov}(f)=\Phi\), \(\operatorname{Cov}(\varepsilon)=\Psi\), and \(\operatorname{Cov}(f,\varepsilon)=0\). The usual factor-analysis restriction is that \(\Psi\) is diagonal, with nonnegative uniqueness variances. The factor covariance matrix \(\Phi\) must be positive definite. These assumptions imply that the model-implied covariance is \(\Sigma=\Lambda\Phi\Lambda^\mathsf{T}+\Psi\).
The transformation changes coordinates in the latent factor space, not the observed variables or their model-implied covariance. The residual covariance \(\Psi\) is left unchanged. Thus a rotation can change which entries of the loadings appear large without changing the total fitted covariance. Factor signs and factor order are especially simple cases of this indeterminacy: reversing one factor’s sign and its loading column, or permuting factor columns and coordinates together, leaves the covariance unchanged.
Orthogonal Rotations
In an orthogonal factor model, \(\Phi=I\), so the factors are uncorrelated and have unit variances. An orthogonal rotation uses a matrix \(T\) satisfying \(T^\mathsf{T}T=I\). Since \(T^{-1}=T^\mathsf{T}\), the rotated factor covariance remains \(I\), and the new loading matrix is \(\Lambda^*=\Lambda T\). The columns of the loadings change, but their row-wise inner products—and therefore the common covariance—do not.
Worked Example: Checking an Orthogonal Rotation
Suppose three standardized profile measurements are represented by two uncorrelated unit-variance factors, with residual covariance \(\Psi=\operatorname{diag}(0.36,0.50,0.60)\), and \(\Lambda=\begin{pmatrix}0.8&0\\0.1&0.7\\0.6&0.2\end{pmatrix}\). Use the orthogonal rotation \(T=\begin{pmatrix}0.8&-0.6\\0.6&0.8\end{pmatrix}\). Its columns have squared lengths \(0.8^2+0.6^2=1\) and dot product \(0.8(-0.6)+0.6(0.8)=0\), so \(T^\mathsf{T}T=I\).
Multiply each loading row by \(T\): \(\Lambda^*=\Lambda T=\begin{pmatrix}0.64&-0.48\\0.50&0.50\\0.60&-0.20\end{pmatrix}\). For example, the second row is \((0.1,0.7)T=(0.1(0.8)+0.7(0.6),\,0.1(-0.6)+0.7(0.8))=(0.50,0.50)\).
With orthogonal factors, the common covariance is \(\Lambda\Lambda^\mathsf{T}\). Before rotation, the off-diagonal entries are \(0.8(0.1)+0(0.7)=0.08\), \(0.8(0.6)+0(0.2)=0.48\), and \(0.1(0.6)+0.7(0.2)=0.20\). After rotation they are \(0.64(0.50)+(-0.48)(0.50)=0.08\), \(0.64(0.60)+(-0.48)(-0.20)=0.48\), and \(0.50(0.60)+0.50(-0.20)=0.20\). The diagonal common variances also agree: they are \(0.64, 0.50,\) and \(0.40\) both before and after rotation. Adding \(\Psi\) therefore gives the same full fitted covariance matrix.
The loadings nevertheless look different. In the original representation the first variable loads on only the first factor; after rotation, it has loadings \(0.64\) and \(-0.48\). Neither representation is distinguished by the fitted covariance alone.
A rotation criterion can select a representation that is easier to describe—for example, one in which each variable has a few large loadings and many small ones. Such a criterion chooses among equivalent representations according to an additional preference. It does not establish that the selected axes are uniquely true latent causes.
Oblique Rotations and Correlated Factors
An oblique transformation allows the factors to be correlated. If the original factors have \(\Phi=I\), a general invertible \(T\) gives \(\Phi^*=T^{-1}T^{-\mathsf{T}}\), which need not be \(I\). The loading matrix and factor covariance must be interpreted together: using \(\Lambda^*\Lambda^{*\mathsf{T}}\) alone after an oblique rotation would generally give the wrong common covariance.
Worked Example: Correcting the Loading Calculation Under an Oblique Transformation
Take the loading matrix \(\Lambda=\begin{pmatrix}0.8&0\\0&0.7\\0.6&0.7\end{pmatrix}\) with orthogonal unit-variance factors, and use the invertible, nonorthogonal transformation \(T=\begin{pmatrix}1&0\\-0.75&1\end{pmatrix}\). The transformed loading matrix is \(\Lambda^*=\Lambda T\). Its second row is calculated as \((0,0.7)T=(0(1)+0.7(-0.75),\,0(0)+0.7(1))=(-0.525,0.7)\). In particular, the second entry is \(0.7\), not \(-0.525\).
The other rows are \((0.8,0)T=(0.8,0)\) and \((0.6,0.7)T=(0.6-0.525,\,0+0.7)=(0.075,0.7)\). Thus \(\Lambda^*=\begin{pmatrix}0.8&0\\-0.525&0.7\\0.075&0.7\end{pmatrix}\).
Here \(T^{-1}=\begin{pmatrix}1&0\\0.75&1\end{pmatrix}\), so the new factor covariance is \(\Phi^*=T^{-1}T^{-\mathsf{T}}=\begin{pmatrix}1&0.75\\0.75&1.5625\end{pmatrix}\). It is positive definite: its first diagonal entry is positive and its determinant is \(1(1.5625)-0.75^2=1\). The covariance between the first and second measurements is zero in the original model, since \((0.8,0)(0,0.7)^\mathsf{T}=0\). In the transformed model it is \((0.8,0)\Phi^*(-0.525,0.7)^\mathsf{T}=0.8[-0.525+0.75(0.7)]=0\). For measurements two and three, the original common covariance is \(0(0.6)+0.7(0.7)=0.49\). The transformed value is \((-0.525,0.7)\Phi^*(0.075,0.7)^\mathsf{T}\). First, \(\Phi^*(0.075,0.7)^\mathsf{T}=(0.6,1.15)^\mathsf{T}\), so the product is \((-0.525)(0.6)+0.7(1.15)=-0.315+0.805=0.49\). This checks covariance preservation while retaining the corrected loading entry.
For oblique solutions, two loading summaries answer different questions. The pattern matrix is the transformed coefficient matrix \(\Lambda^*\): its entries describe the direct contributions of the factors in the model equation. The structure matrix is \(\Lambda^*\Phi^*\); when observed variables and factors are standardized, its entries are correlations between observed variables and factors. With correlated factors, pattern coefficients and structure correlations need not coincide.
Worked Example: Pattern Loadings Versus Structure Correlations
Suppose standardized variables have pattern matrix \(P=\begin{pmatrix}0.8&0\\0.1&0.7\\0.6&0.2\end{pmatrix}\) and factor correlation matrix \(\Phi=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\). The structure matrix is \(P\Phi\). Its rows are \((0.8,0)\Phi=(0.8,0.4)\), \((0.1,0.7)\Phi=(0.45,0.75)\), and \((0.6,0.2)\Phi=(0.7,0.5)\).
For the second variable, the pattern coefficients are \(0.1\) and \(0.7\), while its correlations with factors one and two are \(0.45\) and \(0.75\). The difference arises because factor two is correlated \(0.5\) with factor one. Reporting the pattern entries as if they were correlations would therefore misstate the relationship. Conversely, reporting the structure correlations as direct regression-like contributions would also be misleading.
Why the Invariance Holds
The central result follows directly by substituting the transformed quantities into the common covariance. This argument applies to orthogonal and oblique transformations, provided \(T\) is invertible and the transformed factor covariance is carried along.
Set \(f^*=T^{-1}f\), so \(f=Tf^*\). Substitution into \(x=\mu+\Lambda f+\varepsilon\) gives \(x=\mu+\Lambda T f^*+\varepsilon\), hence \(\Lambda^*=\Lambda T\).
Since \(f^*=T^{-1}f\), its covariance is \(\Phi^*=T^{-1}\Phi T^{-\mathsf{T}}\).
\(\Lambda^*\Phi^*\Lambda^{*\mathsf{T}}=(\Lambda T)(T^{-1}\Phi T^{-\mathsf{T}})(\Lambda T)^\mathsf{T}=\Lambda TT^{-1}\Phi T^{-\mathsf{T}}T^\mathsf{T}\Lambda^\mathsf{T}=\Lambda\Phi\Lambda^\mathsf{T}\), because \(TT^{-1}=I\) and \(T^{-\mathsf{T}}T^\mathsf{T}=I\).
With the same \(\Psi\), the full covariance remains \(\Lambda^*\Phi^*\Lambda^{*\mathsf{T}}+\Psi=\Lambda\Phi\Lambda^\mathsf{T}+\Psi\).
The proof is an exact algebraic identity; it does not depend on a particular data set or estimation algorithm. It does depend on changing the factor covariance consistently. In an orthogonal rotation, the special choice \(T^\mathsf{T}T=I\) keeps \(\Phi=I\). An arbitrary oblique transformation generally does not.
Choosing and Assessing a Rotation
Orthogonal and oblique rotations make different structural commitments. An orthogonal solution maintains uncorrelated factors, which can make the factor axes and summaries simpler. An oblique solution permits correlated factors, often a more plausible description when biological processes overlap. Neither choice is universally preferable: it should reflect the scientific aims and be reported explicitly.
| Feature | Orthogonal rotation | Oblique rotation |
|---|---|---|
| Factor covariance | Identity matrix; factors remain uncorrelated with unit variances | Estimated or specified nonidentity covariance; factors may be correlated |
| What is rotated | Loading axes, using an orthogonal \(T\) | Loading coordinates and factor covariance, using an invertible nonorthogonal \(T\) |
| Loading interpretation | Loadings relate variables to uncorrelated factors; with standardization, they are factor-variable correlations | Distinguish pattern coefficients from structure correlations |
| Main caution | Forcing zero factor correlations may be unrealistic | Ignoring \(\Phi\) or conflating pattern and structure entries can distort interpretation |
In practice, inspect whether the chosen rotation produces a coherent loading pattern, whether the estimated factor correlations are substantial, and whether a few observations or variables dominate the interpretation. Also check that the transformed factor covariance is valid and that the reconstruction \(\Lambda\Phi\Lambda^\mathsf{T}+\Psi\) agrees with the intended fitted covariance. Compare rotated solutions through their fitted covariance or reconstruction, not by demanding identical columns of loadings. Loadings can differ because of rotation, signs, or factor order even when the fit agrees.
A common mistake is to say, “The rotated solution has different loadings, so it must fit the data differently.” That conclusion ignores the factor covariance and the coordinate transformation. The correction is to compare the full common covariance \(\Lambda\Phi\Lambda^\mathsf{T}\), and then the full covariance including \(\Psi\). A related mistake is to assume that a rotation has discovered a unique biological mechanism. A rotation selects a useful representation under a chosen criterion; covariance equivalence alone cannot establish a unique biological meaning for each factor.
Rotational indeterminacy is a limit on interpretation, not a failure of the model. Scientific conclusions that depend on a particular factor orientation should be treated cautiously, especially if plausible rotations substantially change which variables appear to define a factor. Report the rotation type and criterion, the loading matrix, and—when factors are oblique—the factor correlations. These details let readers assess the representation without mistaking it for a uniquely determined result.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Why does an orthogonal transformation preserve \(\Lambda\Lambda^\mathsf{T}\)? State the defining condition on its matrix.
- If an oblique transformation is applied to an orthogonal factor solution, why is it generally incorrect to calculate the new common covariance as \(\Lambda^*\Lambda^{*\mathsf{T}}\)?
- For pattern row \((0.4,0.6)\) and factor correlation matrix \(\begin{pmatrix}1&0.25\\0.25&1\end{pmatrix}\), calculate the corresponding structure row.
- Two rotated solutions have different loading columns. What should be compared to determine whether they imply the same fitted covariance?
Solutions. (1) If \(T^\mathsf{T}T=I\), then \((\Lambda T)(\Lambda T)^\mathsf{T}=\Lambda TT^\mathsf{T}\Lambda^\mathsf{T}=\Lambda\Lambda^\mathsf{T}\). (2) An oblique transformation generally changes the factor covariance to \(\Phi^*=T^{-1}T^{-\mathsf{T}}\), so the common covariance is \(\Lambda^*\Phi^*\Lambda^{*\mathsf{T}}\), not \(\Lambda^*\Lambda^{*\mathsf{T}}\). (3) Multiply the row by the matrix: \((0.4,0.6)\Phi=(0.4+0.15,\,0.10+0.6)=(0.55,0.70)\). (4) Compare \(\Lambda\Phi\Lambda^\mathsf{T}\) for the common covariance and, if the residual covariance is included, compare \(\Lambda\Phi\Lambda^\mathsf{T}+\Psi\). Different loading columns alone do not show that the fitted covariance differs.