Deriving Pairwise Varimax Updates

Why Derive a Pairwise Update?

Varimax searches over orthogonal rotations of a loading matrix for a representation in which each factor has a more differentiated pattern of loadings across variables. A direct search over all possible rotations can be difficult when there are several factors. A pairwise update instead holds all but two columns fixed and chooses the best rotation within that two-column plane. Its key advantage is that the relevant objective reduces to a function of one angle, which can be maximized explicitly.

We derive the update for the raw, unnormalized varimax criterion. If Kaiser row normalization is used, the same derivation applies to the row-normalized loading matrix; the loadings are then returned to their original scale. The choice matters because row normalization changes the matrix to which the criterion is applied.

Definition: For a loading matrix $L=(l_{ij})$ with $p$ rows and $k$ columns, define the raw varimax criterion by $$ V(L)=\sum_{j=1}^{k}\left\{\sum_{i=1}^{p}l_{ij}^{4} -\frac{1}{p}\left(\sum_{i=1}^{p}l_{ij}^{2}\right)^{2}\right\}. $$ A pairwise update replaces two columns by an orthogonal rotation of those columns, choosing the angle that maximizes $V$ while the other columns remain fixed.

Deriving the Two-Column Objective

Let the two columns being updated be $a=(a_1,\ldots,a_p)^\mathsf{T}$ and $b=(b_1,\ldots,b_p)^\mathsf{T}$. Use the rotation convention

a_\theta=a\cos\theta+b\sin\theta,\qquad b_\theta=-a\sin\theta+b\cos\theta.

All other columns contribute a constant to $V$ during this update. For each row define

x_i=a_i^2-b_i^2,\qquad y_i=2a_i b_i.

The squared difference between the two rotated loadings in row $i$ is

a_{\theta i}^{2}-b_{\theta i}^{2} =x_i\cos(2\theta)+y_i\sin(2\theta).

Also, $a_{\theta i}^{2}+b_{\theta i}^{2}=a_i^2+b_i^2$, which does not depend on the angle. Use the identity $u^4+v^4=\{(u^2+v^2)^2+(u^2-v^2)^2\}/2$ to write the fourth-power part of the two-column criterion. For its centering term, the sum of the two column sums of squares is fixed, while their difference is $\sum_i(a_{\theta i}^2-b_{\theta i}^2)$. These facts give the angle-dependent part of the criterion:

V_{\mathrm{pair}}(\theta) =\text{constant}+\frac{1}{2}\left[ \sum_{i=1}^{p}d_i(\theta)^2 -\frac{1}{p}\left(\sum_{i=1}^{p}d_i(\theta)\right)^2 \right], \qquad d_i(\theta)=x_i\cos(2\theta)+y_i\sin(2\theta).

The subtraction involving the sum of the $d_i$ is essential: it comes from the column-mean correction in varimax. Define the symmetric matrix

M= \begin{pmatrix} \sum_i x_i^2-\dfrac{(\sum_i x_i)^2}{p} & \sum_i x_i y_i-\dfrac{(\sum_i x_i)(\sum_i y_i)}{p} \\[6pt] \sum_i x_i y_i-\dfrac{(\sum_i x_i)(\sum_i y_i)}{p} & \sum_i y_i^2-\dfrac{(\sum_i y_i)^2}{p} \end{pmatrix}.

Writing $u=(\cos(2\theta),\sin(2\theta))^\mathsf{T}$, the angle-dependent term is $\tfrac12 u^\mathsf{T}Mu$. Since $u$ has unit length, maximizing the pairwise criterion amounts to choosing a unit vector in the direction of a largest-eigenvalue eigenvector of $M$.

Theorem: For fixed columns $a,b$, a maximizing pairwise angle for the raw varimax criterion is $$ \theta_*=\frac{1}{4}\operatorname{atan2}\!\left(2M_{12},\,M_{11}-M_{22}\right), $$ with equivalent maximizing angles allowed. Here $\operatorname{atan2}(y,x)$ is the quadrant-aware angle of the vector $(x,y)$. If $M$ is a scalar multiple of the identity, the pairwise objective is constant in $\theta$, so every angle maximizes it.

To verify the formula, expand the quadratic form using double-angle identities:

\frac12u^\mathsf{T}Mu =\frac{M_{11}+M_{22}}{4} +\frac{M_{11}-M_{22}}{4}\cos(4\theta) +\frac{M_{12}}{2}\sin(4\theta).

The last two terms have the form $A\cos(4\theta)+B\sin(4\theta)$, with $A=(M_{11}-M_{22})/4$ and $B=M_{12}/2$. Their maximum occurs when the angle $4\theta$ points in the direction $(A,B)$, giving the stated $\operatorname{atan2}$ expression. Equivalently, $(\cos(2\theta),\sin(2\theta))^\mathsf{T}$ is a unit eigenvector for the largest eigenvalue of $M$. If $M$ is scalar, the quadratic form is the same for every unit vector, proving the final claim. $\square$

Worked Updates

Worked Example: Undoing a 30-Degree Mixture

Suppose the two columns are

a=\begin{pmatrix}\sqrt{3}/2\\ \sqrt{3}/2\\ 1/2\end{pmatrix}, \qquad b=\begin{pmatrix}-1/2\\ -1/2\\ \sqrt{3}/2\end{pmatrix}.

These are a 30-degree rotation of the simpler columns $(1,1,0)^\mathsf{T}$ and $(0,0,1)^\mathsf{T}$. Calculate $x_i=a_i^2-b_i^2$ and $y_i=2a_i b_i$:

x=(1/2,1/2,-1/2)^\mathsf{T},\qquad y=(-\sqrt{3}/2,-\sqrt{3}/2,\sqrt{3}/2)^\mathsf{T}.

Their means are $1/6$ and $-\sqrt{3}/6$. Subtracting the corresponding mean products from the sums of products gives

M_{11}=\frac{2}{3},\qquad M_{22}=2,\qquad M_{12}=-\frac{2\sqrt{3}}{3}.

Thus $2M_{12}=-4\sqrt{3}/3$ and $M_{11}-M_{22}=-4/3$. The quadrant-aware angle is $-2\pi/3$, so

\theta_*=\frac14\left(-\frac{2\pi}{3}\right)=-\frac{\pi}{6}.

Applying this rotation gives $a_{\theta_*}=(1,1,0)^\mathsf{T}$ and $b_{\theta_*}=(0,0,1)^\mathsf{T}$. Check the criterion directly. Before rotation, the sums of fourth powers are $19/16+11/16=15/8$, and the column sums of squares are $7/4$ and $5/4$. Therefore $V=15/8-\{(7/4)^2+(5/4)^2\}/3=1/3$. After rotation, the sums of fourth powers total $3$, and the column sums of squares are $2$ and $1$; hence $V=3-(2^2+1^2)/3=4/3$. The update increases the criterion by $1$.

Worked Example: A 45-Degree Update Produces Sparse Columns

Take $p=2$, $a=(1,1)^\mathsf{T}$, and $b=(1,-1)^\mathsf{T}$. Then $x=(0,0)^\mathsf{T}$ and $y=(2,-2)^\mathsf{T}$, so $M_{11}=0$, $M_{22}=8$, and $M_{12}=0$. The formula gives

\theta_*=\frac14\operatorname{atan2}(0,-8)=\frac{\pi}{4}.

The rotated columns are $a_{\theta_*}=(\sqrt{2},0)^\mathsf{T}$ and $b_{\theta_*}=(0,-\sqrt{2})^\mathsf{T}$. Initially, the total fourth-power sum is $4$, and the centering correction is $\{2^2+2^2\}/2=4$, giving $V=0$. After rotation, the fourth-power sum is $8$, while the correction remains $4$, giving $V=4$. The formula has concentrated each column's nonzero loading in a different row.

Worked Example: Equivalent Angles and a Column Swap

Let $a=(1,0)^\mathsf{T}$ and $b=(0,1)^\mathsf{T}$, again with $p=2$. Here $x=(1,-1)^\mathsf{T}$, $y=(0,0)^\mathsf{T}$, and $M$ has diagonal entries $2,0$ and off-diagonal entry $0$. The formula selects $\theta_*=0$. The angle-dependent term is proportional to $\cos^2(2\theta)$, which is also maximized at $\theta=\pi/2$. At that angle the columns are exchanged, with a sign change on one of them. The criterion is unchanged because it treats columns symmetrically and uses only their even powers. This illustrates why the maximizing angle need not be unique even when $M$ is not scalar.

Repeated Pairwise Updates

A typical implementation selects a pair of columns, computes $M$ from their current loadings, applies a maximizing angle, and then moves to another pair. A sweep is one pass through a chosen list of column pairs. At each update, every column outside the selected pair is fixed, so the derivation above applies exactly.

The pairwise update has a useful monotonicity guarantee. For any finite loading matrix, each update leaves the other columns' contributions to $V$ unchanged and maximizes the contribution that can change. Therefore the new value of $V$ is at least its value before the update. Moreover, $V$ is continuous in the entries of $L$, and orthogonal rotation matrices form a closed and bounded set in finite dimensions. Consequently, $V$ is bounded above over rotations of a fixed loading matrix. A sequence of maximizing pair updates therefore produces a nondecreasing, bounded sequence of criterion values, which converges.

This argument concerns the criterion values, not necessarily the loading matrices: different angles can yield equivalent values, and convergence of the values alone does not establish convergence of the matrices. Nor does pairwise maximization guarantee a global maximum over all possible rotations. It is a coordinate-wise strategy whose result can depend on initialization and the order of updates.

Select two columns.
Keep the other columns fixed and form $x_i=a_i^2-b_i^2$ and $y_i=2a_i b_i$ for each row.

Calculate the centered matrix.
Compute $M_{11}$, $M_{12}$, and $M_{22}$, including the subtraction of the column means implied by the $1/p$ terms.

Choose and apply the angle.
Use $\theta_*=\tfrac14\operatorname{atan2}(2M_{12},M_{11}-M_{22})$, then rotate the pair using the stated sign convention.

Continue and assess.
Repeat over pairs as desired, checking that the criterion does not decrease. A stable sequence of criterion values is not proof of a global optimum.

Common Pitfall: Dropping the Centering Correction

A tempting shortcut is to use only $\sum_i x_i^2$, $\sum_i x_i y_i$, and $\sum_i y_i^2$ when building $M$. That shortcut maximizes the fourth-power portion of the objective but ignores the subtraction $\{(\sum_i l_{ij}^2)^2\}/p$ in the varimax criterion. Unless both sums $\sum_i x_i$ and $\sum_i y_i$ happen to be zero, the resulting angle can differ from the varimax update. Include all three mean corrections in $M$.

A second implementation error is to compute a one-argument arctangent of the ratio $2M_{12}/(M_{11}-M_{22})$. That loses quadrant information and can select a minimizing direction rather than a maximizing one. Use $\operatorname{atan2}$, and keep the factor $1/4$: the squared-loadings identities introduce $2\theta$, and maximizing the resulting quadratic introduces $4\theta$.

Key takeaway: The exact two-column varimax update is a two-dimensional eigenvector problem. Its closed-form angle depends on a centered matrix of rowwise squared-loading contrasts; repeated pair updates improve the criterion monotonically, but do not by themselves establish a global solution.

Check Your Understanding

Use the rotation convention and raw varimax criterion stated above.

Why does the pairwise objective depend on $2\theta$ before its quadratic expansion, and on $4\theta$ after expansion?
What terms are lost if the column-mean correction is omitted from $M$?
For what type of matrix $M$ is every pairwise angle maximizing?
Why is $\operatorname{atan2}$ safer than an ordinary arctangent in the update formula?
What does monotone convergence of the criterion values establish, and what does it not establish about the rotated loading matrices?