Target Rotation and Procrustes Alignment

Why Target Rotation and Alignment Are Useful

A factor solution can have loadings that are mathematically equivalent to another solution but arranged in a different orientation. When a hypothesized pattern is available, target rotation chooses an orientation that stays close to that pattern. When two loading matrices need to be compared, orthogonal Procrustes alignment chooses the orientation that makes their entries as close as possible in squared Euclidean distance.

Let $\Lambda$ be a $p\times m$ loading matrix, with $p$ variables and $m$ factors. An orthogonal rotation is represented by an $m\times m$ matrix $Q$ satisfying $Q^TQ=I_m$; the rotated loadings are $\Lambda Q$. The set of such matrices is denoted $O(m)$. The order of multiplication matters: this tutorial uses $\Lambda Q$ throughout.

Definition: Given a target matrix $T$ of the same dimensions as $\Lambda$, and nonnegative weights $w_{ij}$, the weighted target-rotation criterion is $$ F(Q)=\sum_{i=1}^{p}\sum_{j=1}^{m}w_{ij}\bigl((\Lambda Q)_{ij}-T_{ij}\bigr)^2,\qquad Q\in O(m). $$ A zero weight leaves that entry unconstrained by the target. A larger weight penalizes discrepancy at that entry more heavily. Target rotation seeks a $Q$ minimizing $F(Q)$.

The target $T$ need not be an estimated loading matrix. It may encode a hypothesized pattern, such as approximate simple structure, while entries with no prior specification can receive weight zero. The criterion is a preference over orientations, not by itself a test of whether the factor model is correct.

Evaluating a Weighted Target

The weights apply to the squared residuals entry by entry. Thus the residual must first be calculated as $(\Lambda Q)_{ij}-T_{ij}$, and only then squared and multiplied by its weight. A candidate rotation can be assessed directly even when the minimizer has not been found.

Worked Example: Checking a Candidate Against a Weighted Target

Take $\Lambda=\begin{pmatrix}0.8&0\\0.2&1\end{pmatrix}$, $T=\begin{pmatrix}1&1\\0&1\end{pmatrix}$, and weights $W=(w_{ij})=\begin{pmatrix}1&0.1\\1&2\end{pmatrix}$. Evaluate the criterion at $Q=I_2$.

Since $\Lambda I_2=\Lambda$, the residual matrix is \[ \Lambda-T= \begin{pmatrix}-0.2&-1\\0.2&0\end{pmatrix}. \] Consequently, \[ F(I_2)=1(-0.2)^2+0.1(-1)^2+1(0.2)^2+2(0)^2 =0.04+0.10+0.04+0=0.18. \] The entry with residual $-1$ contributes only $0.10$, because its weight is $0.1$. This is the criterion value at the identity, not evidence that the identity is the minimizing rotation.

Worked Example: A Different Target Gives a Different Residual

Now let $\Lambda=I_2$ and $T=\begin{pmatrix}0.8&0\\0.2&1\end{pmatrix}$, with the same weight matrix $W=\begin{pmatrix}1&0.1\\1&2\end{pmatrix}$. Evaluate at $Q=I_2$.

Here $\Lambda Q=I_2$, so the residual is \[ I_2-T=\begin{pmatrix}0.2&0\\-0.2&0\end{pmatrix}. \] It follows that \[ F(I_2)=1(0.2)^2+0.1(0)^2+1(-0.2)^2+2(0)^2 =0.04+0+0.04+0=0.08. \] In particular, the lower-right residual is $1-1=0$, not $-1$. Keeping the target and rotated matrix distinct entry by entry prevents this kind of subtraction error.

What General Weights Do to the Optimization

Because $O(m)$ is a closed and bounded subset of the finite-dimensional space of $m\times m$ matrices, it is compact. The criterion $F(Q)$ is a continuous function of the entries of $Q$. Therefore, for any fixed finite target and nonnegative finite weights, $F$ attains a minimum on $O(m)$. The minimizer need not be unique: zero weights or symmetries in the target can leave multiple orientations equally good.

Entry-specific weights usually prevent a direct reduction to the ordinary Procrustes formula. In the unweighted case, the squared norm of $\Lambda Q$ is constant over orthogonal $Q$, which is the key simplification. With different weights on different entries, the weighted squared norm of $\Lambda Q$ can itself vary with $Q$. The resulting weighted criterion generally needs a numerical constrained optimization method; one should not simply insert weighted cross-products into the unweighted formula.

Worked Example: Weights Can Change the Preferred Orientation

Let $\Lambda=I_2$, use the rotation $Q=\begin{pmatrix}c&-s\\s&c\end{pmatrix}$ with $c^2+s^2=1$, and set $T=\begin{pmatrix}1&1\\0&0\end{pmatrix}$, $W=\begin{pmatrix}1&10\\1&1\end{pmatrix}$. The criterion is \[ F(Q)=(c-1)^2+10(-s-1)^2+s^2+c^2. \] Expanding each term gives \[ F(Q)=2c^2+11s^2+11-2c+20s =13+9s^2-2c+20s, \] where the last equality uses $c^2=1-s^2$. The $9s^2$ term must remain: dropping it would change the optimization problem.

To locate the minimum, write $c=\cos\theta$, $s=\sin\theta$. Differentiation gives \[ \frac{dF}{d\theta}=18\sin\theta\cos\theta+2\sin\theta+20\cos\theta. \] Solving this equation over a full turn and comparing the stationary values gives a minimum at approximately $c=0.4522,\ s=-0.8919$, or $\theta\approx-63.1^\circ$. Substitution yields $F(Q)\approx1.42$. The identity orientation, $c=1,s=0$, has $F(I_2)=11$. This example shows concretely that the weights and the target jointly determine the preferred orientation.

Orthogonal Procrustes Alignment

For alignment, the target is another loading matrix $B$ of the same dimensions as a loading matrix $A$. The unweighted orthogonal Procrustes problem is \[ \min_{Q\in O(m)}\|AQ-B\|_F^2, \] where the Frobenius norm is the square root of the sum of the squared matrix entries. The solution aligns the columns of $A$ to those of $B$ as closely as an orthogonal change of factor coordinates allows.

Theorem: Let $A,B$ be real $p\times m$ matrices, and let $A^TB=U\Sigma V^T$ be a singular value decomposition, with $U,V$ orthogonal and $\Sigma$ diagonal with nonnegative entries. Then $$ Q_* = UV^T $$ minimizes $\|AQ-B\|_F^2$ over $Q\in O(m)$. If $A^TB$ is rank deficient, this formula still gives a minimizer, but the minimizer may not be unique.

Proof. Expand the squared norm using the trace: \[ \|AQ-B\|_F^2 =\operatorname{tr}(Q^TA^TAQ)+\operatorname{tr}(B^TB) -2\operatorname{tr}(Q^TA^TB). \] Since $QQ^T=I_m$, cyclic invariance of trace gives $\operatorname{tr}(Q^TA^TAQ)=\operatorname{tr}(A^TA)$, which does not depend on $Q$. Thus minimizing the distance is equivalent to maximizing $\operatorname{tr}(Q^TC)$, where $C=A^TB$.

Use the stated SVD $C=U\Sigma V^T$, and write $Q=URV^T$. As $Q$ ranges over $O(m)$, so does $R=U^TQV$. Then \[ \operatorname{tr}(Q^TC)=\operatorname{tr}(R^T\Sigma) =\sum_{j=1}^m \sigma_jR_{jj} \leq \sum_{j=1}^m\sigma_j, \] because each diagonal entry of an orthogonal matrix is at most $1$, and each singular value $\sigma_j$ is nonnegative. Equality is attained at $R=I_m$, which corresponds to $Q=UV^T$. This proves the claim. If some singular values vanish, directions in the corresponding null spaces can allow more than one maximizing $R$. $\square$

Worked Example: Aligning Two Loading Matrices

Consider \[ A=\begin{pmatrix}1&0\\0&1\\1&0\\0&1\end{pmatrix}, \qquad B=\begin{pmatrix}3&1\\0&2\\3&1\\0&2\end{pmatrix}. \] The cross-product is \[ A^TB=\begin{pmatrix}6&2\\0&4\end{pmatrix} =2\begin{pmatrix}3&1\\0&2\end{pmatrix}. \] The orthogonal polar factor of the second matrix is $Q_*=\frac{1}{\sqrt{26}}\begin{pmatrix}5&1\\-1&5\end{pmatrix}$. Indeed, $Q_*^TQ_*=I_2$, and \[ Q_*^T\begin{pmatrix}3&1\\0&2\end{pmatrix} =\frac{1}{\sqrt{26}}\begin{pmatrix}15&3\\3&11\end{pmatrix}, \] which is symmetric positive definite. Equivalently, the SVD theorem gives this orthogonal polar factor as $UV^T$. The positive scalar 2 in $A^TB$ does not change that factor.

For a numerical check of the minimized distance, $\|A\|_F^2=4$ and $\|B\|_F^2=28$. The maximum trace term is the sum of the singular values of $A^TB$, equal here to $2\sqrt{26}$: for the matrix $M=\begin{pmatrix}3&1\\0&2\end{pmatrix}$, the sum of its singular values is $\sqrt{\|M\|_F^2+2\det(M)}=\sqrt{14+12}=\sqrt{26}$. Hence \[ \min_{Q\in O(2)}\|AQ-B\|_F^2 =4+28-2(2\sqrt{26}) =32-4\sqrt{26}\approx11.60. \] The nonzero residual is expected: the two matrices are not exact orthogonal copies of one another.

How the Two Problems Are Related

Ordinary target rotation with equal weights is a Procrustes problem: set $A=\Lambda$ and $B=T$. The theorem then supplies a minimizing orthogonal rotation from the SVD of $\Lambda^TT$. General weighted target rotation is broader. Its entrywise weights can make the norm term depend on $Q$, so the unweighted Procrustes derivation no longer applies without additional structure.

This distinction matters in practice. Procrustes alignment is useful when comparing two estimated solutions while accounting for arbitrary orthogonal changes of factor coordinates. Target rotation is appropriate when some loading pattern is hypothesized in advance and discrepancies have specified relative importance. A target with carefully chosen weights can express partial prior structure; it should not be described as an ordinary Procrustes alignment unless the criterion actually reduces to the unweighted squared distance.

Common Mistakes

Subtracting in the wrong direction or changing an entry. The residual is $\Lambda Q-T$. Calculate every entry from those two matrices before squaring it.
Dropping a term after using the circle constraint. In the weighted two-factor example, $c^2+s^2=1$ turns $2c^2+11s^2$ into $2+9s^2$, not into a constant.
Applying the SVD solution to arbitrary weights. The Procrustes theorem above is for the unweighted Frobenius distance. Entry-specific weights generally change the optimization structure.
Reversing the alignment convention. The problem here is $AQ$ close to $B$, so the cross-product is $A^TB$ and the solution is $UV^T$ for its SVD $U\Sigma V^T$.
Assuming a unique answer. A rank-deficient cross-product or an incomplete target can leave some orientations undetermined.

Key takeaway: Weighted target rotation minimizes the explicitly weighted residuals from a hypothesized loading pattern. For unweighted alignment, expanding the Frobenius distance reduces the problem to maximizing a trace, and the SVD of the cross-product gives the orthogonal Procrustes solution.

Check Your Understanding

Use the definitions and derivation above to answer the following.

For a weighted target criterion, what does a zero weight on one loading entry mean?
Why can the ordinary Procrustes SVD formula fail to solve a target problem with entry-specific weights?
If $A^TB=U\Sigma V^T$, which orthogonal matrix minimizes $\|AQ-B\|_F^2$?
In the weighted two-factor example, why does the expansion contain a $9s^2$ term?
What condition can make the Procrustes minimizer nonunique?