Equivalent Covariance Representations After Rotation

Why Rotation Leaves the Common Covariance Unchanged

A factor rotation changes the coordinates used to describe common variation, not the common variation itself. This distinction is especially important for an oblique rotation: the rotated factors may be correlated, so the pattern loadings alone do not specify the common covariance. The factor-covariance matrix is part of the representation.

Let \(x\) be a centered \(p\)-dimensional random vector with a \(q\)-factor representation. Write \(P\) for the pattern-loading matrix, \(f\) for the factor vector, and \(\varepsilon\) for the unique part:

x=Pf+\varepsilon.

Assume \(E[f]=0\), \(E[\varepsilon]=0\), \(\operatorname{Cov}(f)=\Phi\), \(\operatorname{Cov}(\varepsilon)=\Psi\), and \(\operatorname{Cov}(f,\varepsilon)=0\). The usual factor model takes \(\Psi\) to be diagonal. The common covariance is the part contributed by the factors; it is not the full covariance of \(x\).

Definition: The common covariance matrix is \(P\Phi P^{\mathsf T}\), where \(P\) is the pattern-loading matrix and \(\Phi\) is the factor-covariance matrix. Under the stated model assumptions, \(\operatorname{Cov}(x)=P\Phi P^{\mathsf T}+\Psi\).

For an orthogonal factor representation, \(\Phi=I\), so the common covariance is \(PP^{\mathsf T}\). For an oblique representation, \(\Phi\) contains the factor variances and covariances. Omitting it generally changes the implied common covariance.

Rotation and Its Covariance-Preserving Transformation

Let \(T\) be any nonsingular \(q\times q\) matrix and define rotated factors \(f^*=T^{-1}f\). Then \(f=T f^*\), so substituting into the model gives \(x=PTf^*+\varepsilon\). The rotated pattern matrix and factor covariance are therefore

P^*=PT,\qquad \Phi^*=T^{-1}\Phi T^{-\mathsf T}.

The transformed factor covariance follows from the covariance rule for a linear transformation: \(\operatorname{Cov}(T^{-1}f)=T^{-1}\Phi(T^{-1})^{\mathsf T}\). These formulas apply whether or not the rotated factors are standardized.

Theorem: If \(T\) is nonsingular, the rotated representation has the same common covariance as the original representation: \(P^*\Phi^*P^{*\mathsf T}=P\Phi P^{\mathsf T}\). If the unique part is unchanged, it also has the same full covariance.

Proof. Substitute the transformed matrices and use \(T^{-\mathsf T}=(T^{\mathsf T})^{-1}\):

\begin{aligned} P^*\Phi^*P^{*\mathsf T} &=(PT)(T^{-1}\Phi T^{-\mathsf T})(PT)^{\mathsf T}\\ &=PTT^{-1}\Phi T^{-\mathsf T}T^{\mathsf T}P^{\mathsf T}\\ &=P\Phi P^{\mathsf T}. \end{aligned}

Here \(TT^{-1}=I\) and \(T^{-\mathsf T}T^{\mathsf T}=I\). Adding the same \(\Psi\) to both common covariance matrices gives the same full covariance. \(\square\)

In a conventional oblique solution, factors are often rescaled to have unit variances, making \(\Phi\) a correlation matrix. A general nonsingular change of coordinates need not preserve unit factor variances; if standardization is required, the factor coordinates and loadings must be rescaled together. The invariant is the product \(P\Phi P^{\mathsf T}\), not any single matrix in that product.

Worked Example: Checking an Oblique Pattern and Factor Covariance

Take two variables and two factors, with pattern matrix and factor-correlation matrix

P= \begin{pmatrix} 0.8&0.2\\ 0.3&0.7 \end{pmatrix}, \qquad \Phi= \begin{pmatrix} 1&0.5\\ 0.5&1 \end{pmatrix}.

The first variable's communality is its row of \(P\), multiplied on both sides by \(\Phi\):

$$ h_1^2 =0.8^2+0.2^2+2(0.8)(0.2)(0.5) =0.64+0.04+0.16 =0.84. $$

Similarly, the second communality is \(h_2^2=0.3^2+0.7^2+2(0.3)(0.7)(0.5)=0.09+0.49+0.21=0.79\). The common covariance between the variables is \(0.8(0.3)+0.2(0.7)+0.5[0.8(0.7)+0.2(0.3)]=0.24+0.14+0.31=0.69\). Thus

P\Phi P^{\mathsf T} = \begin{pmatrix} 0.84&0.69\\ 0.69&0.79 \end{pmatrix}.

If the uniqueness matrix is \(\Psi=\operatorname{diag}(0.16,0.21)\), the full covariance is \(\begin{pmatrix}1&0.69\\0.69&1\end{pmatrix}\). The diagonal entries are one because each uniqueness variance fills the gap between the communality and total variance.

Communalities, Pattern Loadings, and Structure Loadings

The communality of variable \(i\) is the \(i\)th diagonal entry of the common covariance. If \(p_i\) is row \(i\) of \(P\), then

h_i^2=p_i\Phi p_i^{\mathsf T}.

When factors are uncorrelated and have unit variances, \(\Phi=I\), and this reduces to the familiar sum of squared pattern loadings. With correlated factors, cross-products contribute. Thus the sum of squared pattern loadings is generally not the communality after oblique rotation.

The structure matrix records the covariances between observed variables and factors. Under the model assumptions, let \(S=\operatorname{Cov}(x,f)\). Then

S=P\Phi, \qquad P\Phi P^{\mathsf T}=PS^{\mathsf T}.

When the observed variables and factors are standardized, the entries of \(S\) are correlations. The structure matrix is not the pattern matrix: the pattern describes the coefficients multiplying the factors, while the structure describes observed-variable–factor associations. Their distinction matters for calculating communalities.

Result: For each variable, \(h_i^2=p_i\Phi p_i^{\mathsf T}=p_i s_i^{\mathsf T}\), where \(s_i\) is row \(i\) of \(S=P\Phi\). The sum of squared pattern loadings gives the communality when \(\Phi=I\), but not in general.

Derivation. Since \(S=P\Phi\), its \(i\)th row is \(s_i=p_i\Phi\). The factor covariance matrix is symmetric, so \(s_i^{\mathsf T}=\Phi p_i^{\mathsf T}\). Therefore \(p_i s_i^{\mathsf T}=p_i\Phi p_i^{\mathsf T}=h_i^2\). If \(\Phi=I\), this is \(p_i p_i^{\mathsf T}\), the sum of the squared entries in \(p_i\). \(\square\)

Worked Example: Recovering Communalities From the Structure Matrix

Use the matrices in the first example. Compute the structure matrix:

S=P\Phi = \begin{pmatrix} 0.8&0.2\\ 0.3&0.7 \end{pmatrix} \begin{pmatrix} 1&0.5\\ 0.5&1 \end{pmatrix} = \begin{pmatrix} 0.9&0.6\\ 0.65&0.85 \end{pmatrix}.

For the first variable, the dot product of its pattern row and structure row is \(0.8(0.9)+0.2(0.6)=0.72+0.12=0.84\). For the second, it is \(0.3(0.65)+0.7(0.85)=0.195+0.595=0.79\). These match the diagonal entries of \(P\Phi P^{\mathsf T}\).

By contrast, the squared pattern-loading sums are \(0.8^2+0.2^2=0.68\) and \(0.3^2+0.7^2=0.58\), neither of which is the corresponding communality. The missing contributions are the factor-correlation cross-products, \(0.16\) and \(0.21\).

Numerically Verifying Equivalent Rotated Representations

A useful verification checks both the transformed parameters and the resulting covariance entries. It is not enough to observe that the loading matrices differ: different coordinates can represent the same common covariance.

Worked Example: A Nonorthogonal Change of Factor Coordinates

Starting with the matrices above, use the nonsingular transformation

T= \begin{pmatrix} 1&0.4\\ 0&1 \end{pmatrix}, \qquad T^{-1}= \begin{pmatrix} 1&-0.4\\ 0&1 \end{pmatrix}.

The rotated pattern matrix is \(P^*=PT=\begin{pmatrix}0.8&0.52\\0.3&0.82\end{pmatrix}\). The rotated factor covariance is

\Phi^*=T^{-1}\Phi T^{-\mathsf T} = \begin{pmatrix} 0.76&0.10\\ 0.10&1 \end{pmatrix}.

For example, its upper-left entry is \(1-2(0.4)(0.5)+(0.4)^2=0.76\), and its off-diagonal entry is \(0.5-0.4=0.10\). Direct multiplication gives

P^*\Phi^*P^{*\mathsf T} = \begin{pmatrix} 0.84&0.69\\ 0.69&0.79 \end{pmatrix} =P\Phi P^{\mathsf T}.

The entries can also be checked individually. The first diagonal entry is \(0.76(0.8^2)+2(0.10)(0.8)(0.52)+0.52^2=0.4864+0.0832+0.2704=0.84\). The second is \(0.76(0.3^2)+2(0.10)(0.3)(0.82)+0.82^2=0.0684+0.0492+0.6724=0.79\). The off-diagonal entry is \(0.76(0.8)(0.3)+0.10[0.8(0.82)+0.52(0.3)]+0.52(0.82)=0.1824+0.0812+0.4264=0.69\).

Here \(\Phi^*\) is a factor-covariance matrix, not a correlation matrix, because its first diagonal entry is \(0.76\). This does not undermine the covariance equivalence: the transformed coordinates have different factor variances, and their loading and covariance matrices together preserve the same common covariance.

Common Pitfall and Why the Distinction Matters

Wrong reasoning: “The communality is always the sum of the squared pattern loadings, so I can calculate it from the rotated pattern matrix alone.” That rule silently assumes uncorrelated unit-variance factors. Under oblique rotation, the off-diagonal terms of \(\Phi\) contribute to \(p_i\Phi p_i^{\mathsf T}\), and dropping them can give incorrect communalities and an incorrect implied covariance.

The correction is to use the full product \(P\Phi P^{\mathsf T}\). An equivalent calculation uses the pattern and structure matrices: \(PS^{\mathsf T}\). These identities are useful when checking software output, comparing two rotated representations, or interpreting a reported communality. They also clarify what is invariant: the common covariance and its diagonal communalities stay fixed under a valid change of factor coordinates, even though the pattern loadings and factor covariances can change.

Use \(PP^{\mathsf T}\) only when the factors have identity covariance.
For correlated factors, include \(\Phi\) when calculating common covariance or communalities.
Do not interpret a transformed pattern matrix as though it alone encoded the covariance.
When factors are rescaled, check that the factor covariance is transformed along with the loadings.

Check Your Understanding

Use the matrix identities and assumptions stated above to answer the following.

What assumption on \(\Phi\) makes a communality equal to the sum of squared pattern loadings?
For an invertible \(T\), write the transformed pattern matrix and factor covariance that preserve common covariance.
Explain why the pattern matrix and structure matrix differ when factors are correlated.
For row \(p_i\), express its communality using both \(\Phi\) and the corresponding structure row \(s_i\).
If a transformed factor covariance is not a correlation matrix, does that alone show that the common covariance changed? Explain.