Stability of Rotations Under Sampling and Perturbations

Why Rotation Stability Needs Its Own Check

A rotated factor solution is an interpretation of a fitted model, not just a list of numbers. If small changes to the observations or to the optimizer’s starting values produce substantially different loading patterns, the interpretation may depend on those choices. Stability assessment makes that sensitivity visible.

Two complications must be separated. First, factor labels and signs are arbitrary: exchanging two factors or reversing one factor’s sign can change the displayed loadings without changing the fitted covariance representation. Second, a rotation algorithm can find genuinely different solutions, or small data changes can alter the estimated loadings even after labels are aligned. The first is a representation difference; the second is the instability of interest.

For a \(p\)-variable, \(k\)-factor model, write the common covariance representation as \(\Lambda\Phi\Lambda^\mathsf{T}+\Psi\), where \(\Lambda\) is the loading matrix, \(\Phi\) is the factor-correlation matrix, and \(\Psi\) is the uniqueness matrix. A stability comparison should examine the aligned loading matrices, the corresponding factor correlations, and the substantive conclusions drawn from them.

Align Before Comparing

Suppose two fitted solutions have the same number of factors. A signed permutation matrix \(Q\) is a square matrix with exactly one nonzero entry, either \(+1\) or \(-1\), in each row and column. It represents factor reordering and sign reversal. If \((\Lambda,\Phi)\) is one solution, the equivalent relabelled representation is

For comparing two results, choose a signed permutation that best matches their factor columns. A common approach is to calculate absolute Tucker congruence for every pair of columns, choose the one-to-one pairing that maximizes the total congruence, and then choose each factor’s sign to make its matched congruence nonnegative. This is a discrete alignment; it does not license an arbitrary rotation of one fitted solution to make it resemble another. In particular, rotating away substantive differences would defeat the stability assessment.

Alignment must be applied consistently to \(\Lambda\) and \(\Phi\). Comparing loadings after a factor swap but leaving the factor-correlation matrix in its original order produces mismatched quantities.

Proof. A signed permutation matrix is orthogonal, so \(QQ^\mathsf{T}=I\). Substitution gives

The transformed \(\Phi_Q\) is still a correlation matrix: its diagonal entries remain one, and its off-diagonal entries are reordered and, where appropriate, sign-reversed. Thus a mere change of factor labels or signs does not change the common covariance representation. \(\square\)

What to Compare After Alignment

No single number captures rotation stability. Useful summaries answer different questions, so report them together with the perturbation scheme and the alignment method.

• Loading changes: compare the aligned loading matrices using entrywise changes, the maximum absolute change, or a matrix distance such as the Frobenius norm. A distance should be interpreted relative to the loading scale and the number of entries.
• Factor correlations: compare aligned \(\Phi\) matrices. Changes in correlations can affect interpretation even when each factor’s largest loadings look similar.
• Salient-variable assignments: specify a threshold or a dominant-loading rule, then check whether variables retain their assignments across solutions. Report ambiguous or changing assignments rather than hiding them in a single summary.
• Rotation criterion and convergence: record criterion values and whether runs converged. Similar criterion values do not guarantee similar loading patterns, and a small loading difference does not by itself establish equivalent interpretations.

For sampling perturbations, one can compare solutions from resamples or other repeated samples. The earlier material on Bootstrap Uncertainty for Rotated Loadings treats bootstrap uncertainty; here the additional point is to use the resulting aligned solutions to assess the stability of the interpretation. For optimization sensitivity, rerun the rotation from multiple starting configurations while holding the fitted unrotated information and rotation criterion fixed. These checks address different sources of variability and should not be conflated.

Local Stability of an Optimized Rotation

A useful local result explains when a small change in the data should produce a small change in the fitted rotation. It is a conditional result: the solution must be locally identified, and the criterion must have a nonsingular curvature there. Rotational equivalence, factor collisions, or a flat criterion can violate those conditions.

Proof. Define \(G(\theta,z)=\nabla_\theta J(\theta;z)\). The assumptions imply that \(G\) is continuously differentiable near \((\theta_0,z_0)\), \(G(\theta_0,z_0)=0\), and its derivative with respect to \(\theta\) there is \(H_0\), which is invertible because it is positive definite. The implicit function theorem therefore gives neighborhoods of \(z_0\) and \(\theta_0\), and a unique continuously differentiable function \(\theta(z)\) in those neighborhoods satisfying \(G(\theta(z),z)=0\) and \(\theta(z_0)=\theta_0\).

By continuity of the Hessian, it remains positive definite in a sufficiently small neighborhood of \((\theta_0,z_0)\); hence these stationary solutions are strict local minima. The derivative \(D\theta(z)\) is continuous. Choose a closed ball about \(z_0\) contained in the neighborhood. On this compact ball its operator norm has a finite upper bound \(C\). Applying the mean-value bound along the line segment from \(z_0\) to \(z\) gives \(\|\theta(z)-\theta_0\|\leq C\|z-z_0\|\). This proves local uniqueness, strict local minimality, and the stated stability bound. \(\square\)

This theorem concerns a local solution branch, not a guarantee that a numerical optimizer will find that branch from every start. It also does not apply directly when the parameter is not locally identified. In a factor model, one must account for arbitrary factor labels and signs, and any continuous rotational indeterminacy must be resolved by the criterion or identification constraints before the Hessian can be nonsingular.

Proof. For any entry \(a\) and its perturbed value \(\tilde a\), the reverse triangle inequality gives \(\bigl||\tilde a|-|a|\bigr|\leq|\tilde a-a|\leq\delta\). The dominant absolute loading can therefore decrease by at most \(\delta\), while a competing loading can increase by at most \(\delta\). A gap greater than \(2\delta\) remains positive, so the dominant factor does not change. The dominant loading also remains above \(c\), because it can decrease by at most \(\delta\) and its original margin over \(c\) exceeds \(\delta\). \(\square\)

Common Pitfall and Practical Interpretation

A common mistake is to compare loading columns in their printed order and declare instability when factors have merely been swapped or sign-reversed. The correction is to align factors first, applying the same transformation to \(\Phi\), and then compare the substantive features. The opposite error is also possible: allowing an unrestricted rotation during alignment can make genuinely different solutions look artificially alike. Use only transformations that represent the agreed factor-label and sign indeterminacies for the comparison.

Stability is always relative to the perturbation and to the interpretation rule. A solution may be stable across random starts but sensitive to a few observations, or stable in its largest loadings but not in factor correlations. Report what was perturbed, how many fits were compared, how alignment was performed, and which aspects changed. When salient assignments are close to a threshold or two loadings are nearly tied, describe them as fragile rather than forcing a categorical conclusion.