Orthogonal Rotations and the Varimax Criterion

Why Rotate Factor Loadings?

A factor solution can represent the observed covariance well yet be difficult to interpret: several variables may have moderate loadings on multiple factors. Rotation changes the coordinate system used to describe the common factors, not the fitted covariance representation. The varimax criterion chooses an orthogonal orientation that tends to make each factor's loading pattern more differentiated across variables.

The criterion is a mathematical preference for a particular kind of simplicity. It favors columns in which squared loadings vary substantially from variable to variable, rather than columns whose squared loadings are all similar. The word “tends” matters: maximizing varimax is an optimization rule, not a guarantee that the resulting factors have a unique or substantively correct interpretation.

Orthogonal Rotations

Let \(\Lambda\) be a \(p\times q\) loading matrix for \(p\) observed variables and \(q\) common factors. An orthogonal rotation is determined by a \(q\times q\) matrix \(T\) satisfying \(T'T=TT'=I_q\), where \(I_q\) is the \(q\times q\) identity matrix. The rotated loading matrix is \(\Lambda^*=\Lambda T\).

For example, in the common-factor model \(X=\mu+\Lambda F+\varepsilon\), with \(\operatorname{Cov}(F)=I_q\) and residual covariance matrix \(\Psi\), the rotated representation uses \(\Lambda^*=\Lambda T\) and \(F^*=T'F\). Since \(TT'=I_q\), \(\Lambda^*F^*=\Lambda F\); also, \(\operatorname{Cov}(F^*)=T'I_qT=I_q\). Thus the rotated factors remain uncorrelated with unit variances, and the model-implied covariance is unchanged.

These equalities explain why rotation is a re-expression of a fitted solution rather than a new extraction of factors. What can change is the allocation of a variable's fixed communality among the factor columns. Varimax evaluates that allocation in search of a more interpretable orientation.

Deriving the Varimax Criterion

Write \(\lambda_{ij}\) for the loading of variable \(i\) on factor \(j\). For each factor column, take the variance across the \(p\) variables of its squared loadings, using divisor \(p\). The unnormalized-row version of the varimax criterion is the sum of these column variances.

To derive the displayed expression, define \(x_{ij}=\lambda_{ij}^2\) and let \(\bar{x}_j=(1/p)\sum_i x_{ij}\). The variance in column \(j\) is \[ \frac{1}{p}\sum_i(x_{ij}-\bar{x}_j)^2 =\frac{1}{p}\sum_i x_{ij}^2-\bar{x}_j^2 =\frac{1}{p}\sum_i\lambda_{ij}^4-\left(\frac{1}{p}\sum_i\lambda_{ij}^2\right)^2. \] Summing over the columns gives \(V(\Lambda)\). This derivation also identifies an easy arithmetic trap: the fourth-power term comes from squaring a squared loading, so a loading of \(0.5\) contributes \(0.5^4=0.0625\) there, not \(0.5\).

Why does this variance reflect simplicity? For a single column, the quantities \(x_i=\lambda_{ij}^2\) are nonnegative. If their sum \(s\) is fixed, their variance is \[ \frac{1}{p}\sum_i x_i^2-\left(\frac{s}{p}\right)^2. \] The second term is fixed, so greater variance means a greater sum of squared \(x_i\)'s. Since \(s^2=\sum_i x_i^2+2\sum_{i<k}x_ix_k\), this sum is largest when the \(x_i\)'s are concentrated, and smallest when they are equal. In loading terms, concentration means some variables have large squared loadings on that factor while others have small ones.

There is an important qualification: an orthogonal rotation preserves the total sum of squared loadings across the whole matrix, but it does not generally preserve each column's sum of squares. Therefore the fixed-\(s\) argument explains the column-wise simplicity interpretation; it does not mean that each rotated column has a fixed norm. The criterion is evaluated for the complete rotated matrix.

What Maximization Means

For a fixed initial loading matrix, varimax searches over the set of orthogonal matrices \(T\) and selects an orientation with a large value of \(V(\Lambda T)\). A solution need not be unique: permuting factor columns or changing a column's sign leaves the criterion unchanged. These transformations also do not alter the fitted covariance. More generally, the maximum criterion value exists, even if there are multiple maximizing rotations.

Proof. The set \(O(q)=\{T:T'T=I_q\}\) is closed, because its defining equation is continuous in the entries of \(T\). It is bounded: each column has squared norm \(1\), so every entry has absolute value at most \(1\). A closed and bounded subset of finite-dimensional Euclidean space is compact. The entries of \(\Lambda T\) are linear functions of the entries of \(T\), and \(V\) is a polynomial in the loading entries. Therefore \(T\mapsto V(\Lambda T)\) is continuous. The extreme value theorem gives a maximum on the compact set \(O(q)\). \(\square\)

The criterion is unchanged by column signs because it uses only even powers of the loadings. It is unchanged by column permutations because it sums the same column-wise quantities over all factors. This invariance is useful: varimax is assessing the pattern of loading concentration, not attaching a unique name or sign to each factor.

Interpretation and a Common Pitfall

A high varimax value indicates that, across the variables, squared loadings within each factor column are relatively uneven. A factor may have large loadings for a subset of variables and small loadings for the rest, making a proposed interpretation easier to describe. Since communalities remain fixed, rotation redistributes each variable's common variance across the factors rather than changing how much common variance the model assigns to that variable.

• Wrong reasoning: “The fitted covariance is unchanged by an orthogonal rotation, so the varimax criterion must also be unchanged.”
• Correction: The fitted covariance depends on \(\Lambda\Lambda'\), which is invariant under \(\Lambda\mapsto\Lambda T\). Varimax instead depends on fourth powers and column-wise means of squared loadings, which generally do change with \(T\).
• Wrong reasoning: “A loading of \(0.5\) contributes \(0.5\) to the squared-loading calculation.”
• Correction: Its squared loading is \(0.5^2=0.25\); the fourth-power term uses \((0.5^2)^2=0.0625\). Apply the formula to the loadings in order, without skipping either squaring step.

Varimax is a criterion for choosing a convenient representation, not a test of model fit and not evidence that the rotated factors are uniquely determined by the data. Its value is that it turns the idea of loading simplicity into a precisely defined objective while retaining the covariance representation of the fitted factor model.