Factor Rotation as a Likelihood-Preserving Reparameterization

Why Factor Coordinates Can Change Without Changing the Model

A factor model describes the covariance among observed variables through a smaller collection of latent factors. But the coordinates used to describe those factors are not always uniquely determined by the covariance model. If we change the factor coordinates and adjust the loadings and factor covariance consistently, the same observed covariance can have a different-looking parameter representation. This matters because a change in the displayed loadings does not necessarily mean the fitted model has changed.

Let \(X\) be a \(p\)-dimensional observed random vector and \(F\) a \(q\)-dimensional factor vector, with \(q\leq p\). The common-factor model is

Here \(\Lambda\) is a \(p\times q\) loading matrix, \(\Phi\) is a positive-definite \(q\times q\) factor-covariance matrix, and \(\Psi\) is typically a positive diagonal \(p\times p\) uniqueness matrix. The zero cross-covariance assumption is what removes cross terms when calculating the covariance of \(X\).

The covariance formula follows by expanding \(\operatorname{Cov}(\Lambda F+\varepsilon)\); the cross-covariance terms vanish because \(\operatorname{Cov}(F,\varepsilon)=0\). The central question is how this representation changes when the latent coordinates are replaced by another invertible system.

Invertible Changes of Factor Coordinates

Let \(A\) be any invertible \(q\times q\) matrix. Define new factors by \(F^*=A^{-1}F\). Then \(F=AF^*\), so the same observed vector can be written using new parameters \(\Lambda^*=\Lambda A\) and \(\Phi^*=A^{-1}\Phi A^{-T}\). The uniqueness matrix and mean are left unchanged.

Proof. Since \(A^{-T}=(A^{-1})'\), substitution gives

because \(AA^{-1}=I_q\) and \(A^{-T}A'=I_q\). Also, \(F^*\) has mean zero and covariance \(A^{-1}\Phi A^{-T}\); it remains uncorrelated with \(\varepsilon\). Since \(\Phi\) is positive definite and \(A\) is invertible, \(\Phi^*\) is positive definite as well: for nonzero \(v\), \(v'\Phi^*v=(A^{-T}v)'\Phi(A^{-T}v)>0\). The transformed parameters therefore define a valid covariance representation. \(\square\)

If the factor model is Gaussian, the observed distribution is determined by \(\mu\) and \(\Sigma\). More specifically, for \(n\) independent observations \(x_1,\ldots,x_n\), its observed-data log-likelihood is

When the transformation leaves \(\mu\) and \(\Sigma\) unchanged, it leaves every term in this expression unchanged. Thus the observed-data likelihood is identical at the two representations. This conclusion concerns the likelihood of the observations, not a density for factor coordinates treated as if they were observed.

What Orthogonal Rotations Preserve

An orthogonal matrix \(Q\) satisfies \(Q'Q=QQ'=I_q\). The corresponding rotation has \(\Lambda^*=\Lambda Q\) and \(\Phi^*=Q'\Phi Q\), which is the general transformation above with \(A=Q\). In particular, if factors are standardized and uncorrelated so that \(\Phi=I_q\), then \(\Phi^*=Q'I_qQ=I_q\). Orthogonal rotations therefore change the loading coordinates while keeping the factor covariance equal to the identity.

Proof. The common covariance after rotation is

using \(QQ'=I_q\) twice. The uniqueness matrix is not altered, so the full covariance and each diagonal common-covariance entry are unchanged. If \(\Phi=I_q\), the communality is \((\Lambda\Lambda')_{ii}\), which equals the squared Euclidean length of row \(i\) of \(\Lambda\). Finally, the total common variance is the trace of the common covariance, \(\operatorname{tr}(\Lambda\Phi\Lambda')\), so it too is unchanged. \(\square\)

These invariants describe the fitted covariance structure, not every feature of the loading table. In particular, the loading of a given variable on a particular rotated factor can change, as can the squared norm of an individual loading column. The total common variance is preserved, but its allocation across columns generally is not.

When the Reparameterization Is—and Is Not—Allowed

The scope of the invariance depends on the model’s parameter constraints. If \(\Phi\) is allowed to be any positive-definite matrix, the general invertible transformation preserves the covariance representation. If the model instead fixes \(\Phi=I_q\), a general invertible transformation will usually produce \(\Phi^*\ne I_q\). To retain the identity constraint, it must satisfy \(A^{-1}A^{-T}=I_q\), equivalently \(AA'=I_q\); for a square matrix this means \(A\) is orthogonal.

Other restrictions can also matter. For example, a loading pattern that requires specified entries of \(\Lambda\) to be zero may not retain those zeros after an arbitrary rotation. The unrestricted covariance representations can be equivalent even when one of them fails to satisfy a particular restricted loading pattern. Equivalence of covariances does not automatically establish equivalence within every constrained parameter space.

Common Mistake and Practical Significance

Common mistake: “The loading matrix changed, so the fitted model and likelihood must have changed.” This reasoning confuses parameter coordinates with the observed distribution. The correction is to calculate the transformed factor covariance as well: compare \(\Lambda\Phi\Lambda'\), not \(\Lambda\) alone. If the mean and the full covariance remain the same, the Gaussian observed-data likelihood remains the same too.

This distinction is useful when interpreting fitted factors. Different loading matrices can express the same covariance structure, so a particular loading pattern is not automatically a unique discovery about the data. At the same time, not every numerical transformation is admissible under every identification or loading constraint. Always check both the covariance identity and the parameter-space restrictions before calling two representations equivalent.