Identifying Constraints for Factor Scale and Orientation

Why Factor Constraints Need Careful Interpretation

A factor model describes observed covariance through latent variables, but the latent variables have no directly observed units or axes. Consequently, different numerical loadings and factor covariances can describe exactly the same observed covariance matrix. A normalization can choose convenient units; an orientation restriction can choose particular latent axes. Neither should automatically be interpreted as evidence that the underlying parameters are uniquely determined by the data.

The central distinction is between choosing a representation and identifying a model. A representation constraint selects among equivalent descriptions. Identification asks whether the observable distribution determines the parameter values, subject to the model and its restrictions. This tutorial derives the scale and orientation transformations that create ambiguity and then examines common constraints.

The Transformation Behind the Ambiguity

Definition: In the common-factor model, let $X$ be a $p$-dimensional centered observed vector and write $$ X=\Lambda F+\varepsilon, $$ where $\Lambda$ is a $p\times k$ loading matrix, $F$ is a $k$-dimensional factor vector with mean zero and covariance matrix $\Phi$, and $\varepsilon$ is a $p$-dimensional specific-error vector with mean zero and covariance matrix $\Psi$. Assume $\operatorname{Cov}(F,\varepsilon)=0$. Then the model-implied covariance is $$ \Sigma=\Lambda\Phi\Lambda^{\mathsf T}+\Psi. $$

Suppose $A$ is any nonsingular $k\times k$ matrix. Define new factors $F^*=A^{-1}F$, new loadings $\Lambda^*=\Lambda A$, and new factor covariance $\Phi^*=A^{-1}\Phi A^{-\mathsf T}$, where $A^{-\mathsf T}=(A^{-1})^{\mathsf T}$. The observed vector is unchanged because $\Lambda^*F^*=\Lambda A A^{-1}F=\Lambda F$. The covariance decomposition is unchanged as well:

\begin{aligned} \Lambda^*\Phi^*\Lambda^{*\mathsf T}+\Psi &=\Lambda A(A^{-1}\Phi A^{-\mathsf T})A^{\mathsf T}\Lambda^{\mathsf T}+\Psi\\ &=\Lambda\Phi\Lambda^{\mathsf T}+\Psi\\ &=\Sigma. \end{aligned}

Thus, without restrictions, the same covariance can have many parameter representations. This result is an algebraic invariance, not a claim that every transformed parameter value satisfies every possible restriction. For example, a restriction that $\Phi=I_k$ limits the transformations that remain allowable.

Theorem: If $A$ is nonsingular, the transformation $(\Lambda,\Phi)\mapsto(\Lambda A,A^{-1}\Phi A^{-\mathsf T})$ preserves the common covariance component $\Lambda\Phi\Lambda^{\mathsf T}$. If $\Phi=I_k$ is imposed both before and after transformation, the allowable $A$ are precisely the orthogonal matrices.

Proof. The covariance preservation follows by substitution and cancellation:

(\Lambda A)(A^{-1}\Phi A^{-\mathsf T})(\Lambda A)^{\mathsf T} =\Lambda A A^{-1}\Phi A^{-\mathsf T}A^{\mathsf T}\Lambda^{\mathsf T} =\Lambda\Phi\Lambda^{\mathsf T}.

Now suppose $\Phi=I_k$, and require $\Phi^*=I_k$. Then $A^{-1}A^{-\mathsf T}=I_k$. Multiplying on the left by $A$ and on the right by $A^{\mathsf T}$ gives $AA^{\mathsf T}=I_k$. Since $A$ is square and nonsingular, this implies $A^{-1}=A^{\mathsf T}$, and hence $A^{\mathsf T}A=I_k$. Thus $A$ is orthogonal. Conversely, if $A$ is orthogonal, then $A^{-1}=A^{\mathsf T}$, so $A^{-1}A^{-\mathsf T}=A^{\mathsf T}A=I_k$. Therefore exactly the orthogonal transformations preserve the identity factor covariance constraint. $\square$

Scale Normalization Is Not Orientation Identification

A factor scale is arbitrary because multiplying a factor by a nonzero constant and dividing its loading column by the same constant leaves their product unchanged. A common convention is to give the factors unit variances. In a model with uncorrelated factors, this is expressed as $\Phi=I_k$. It fixes factor units and makes their covariance matrix simple. But, as the theorem shows, it does not determine the factor axes: every orthogonal rotation remains allowable unless additional restrictions are imposed on the loadings.

There is an important distinction between fixing the diagonal entries of $\Phi$ to one and imposing $\Phi=I_k$. The first says each factor has unit variance but permits nonzero factor correlations. It does not, in general, eliminate arbitrary changes of coordinates. For example, starting from $\Phi=I_2$, let

B=\begin{pmatrix}1&0\\ c&\sqrt{1-c^2}\end{pmatrix},\qquad A=B^{-1},\qquad 0<|c|<1.

Under the transformation, $\Phi^*=A^{-1}A^{-\mathsf T}=BB^{\mathsf T}$, whose diagonal entries are both one, while its off-diagonal entries equal $c$. The transformed factors still have unit variances, but they are correlated. Thus, “unit factor variances” alone is not the same restriction as “standardized and uncorrelated factors.” Which convention is appropriate depends on the model being used.

Worked Example: Changing a One-Factor Scale

Let $F$ have variance $4$, let $\Lambda=(1,2)^{\mathsf T}$, and let $\Psi=\operatorname{diag}(1,1)$. The common covariance contribution is

\Lambda\Phi\Lambda^{\mathsf T} = \begin{pmatrix}1\\2\end{pmatrix} (4) \begin{pmatrix}1&2\end{pmatrix} = \begin{pmatrix}4&8\\8&16\end{pmatrix}.

Set $A=2$, so $F^*=F/2$, $\Lambda^*=2\Lambda=(2,4)^{\mathsf T}$, and $\Phi^*=4/4=1$. The new common covariance is

\Lambda^*\Phi^*\Lambda^{*\mathsf T} = \begin{pmatrix}2\\4\end{pmatrix} (1) \begin{pmatrix}2&4\end{pmatrix} = \begin{pmatrix}4&8\\8&16\end{pmatrix}.

The observed covariance, including $\Psi$, is the same in both descriptions. The unit-variance convention changes the factor's units and the numerical loadings, not the covariance model. For one factor, the remaining ambiguity after setting its variance to one includes the sign change $F^*=-F$, $\Lambda^*=-\Lambda$.

Orientation Constraints Choose Axes

When $\Phi=I_k$, orthogonal rotations preserve the covariance. In a two-factor model, a rotation through an angle $\theta$ is

Q(\theta)= \begin{pmatrix} \cos\theta&-\sin\theta\\ \sin\theta&\cos\theta \end{pmatrix}, \qquad Q(\theta)Q(\theta)^{\mathsf T}=I_2.

Replacing $\Lambda$ by $\Lambda Q(\theta)$ gives another loading matrix with the same common covariance. The columns, and therefore the factor axes, change. A loading restriction such as a specified zero or a triangular form can select an orientation from this set. A sign convention can then choose between remaining sign-reversed versions.

Worked Example: A Two-Factor Rotation

Take $\Phi=I_2$, $\Psi=I_3$, and

\Lambda= \begin{pmatrix} 1&0\\ 0&1\\ 1&1 \end{pmatrix}.

The common covariance is

\Lambda\Lambda^{\mathsf T} = \begin{pmatrix} 1&0&1\\ 0&1&1\\ 1&1&2 \end{pmatrix}.

Choose $\theta=45^\circ$, so $c=s=1/\sqrt{2}$. Then

Q= \begin{pmatrix}c&-s\\s&c\end{pmatrix}, \qquad \Lambda Q= \begin{pmatrix} c&-s\\ s&c\\ c+s&c-s \end{pmatrix}.

Since $QQ^{\mathsf T}=I_2$,

(\Lambda Q)(\Lambda Q)^{\mathsf T} =\Lambda QQ^{\mathsf T}\Lambda^{\mathsf T} =\Lambda\Lambda^{\mathsf T}.

For instance, the transformed first and second observed variables have common covariance $cs+(-s)c=0$, just as before; the covariance of the first and third is $c(c+s)+(-s)(c-s)=1$. The loading entries look different, but the observable covariance has not changed. Unit factor variances alone therefore do not give a unique orientation.

Result: Under the restriction $\Phi=I_k$, orthogonal transformations preserve the model-implied covariance. Consequently, fixing factor variances and requiring uncorrelated factors removes general scale-and-coordinate changes but leaves rotational ambiguity in $\Lambda$, unless loading restrictions further constrain it.

Proof. For any orthogonal $Q$, $Q Q^{\mathsf T}=I_k$, so

(\Lambda Q)I_k(\Lambda Q)^{\mathsf T} =\Lambda Q Q^{\mathsf T}\Lambda^{\mathsf T} =\Lambda\Lambda^{\mathsf T}.

The transformed factor covariance remains $I_k$, by the theorem. Hence every such rotation gives the same $\Sigma$. If the loading restrictions admit only some of these rotated matrices, those restrictions reduce the remaining ambiguity; if they admit all of them, the orientation is still undetermined. $\square$

Worked Example: A Triangular Orientation Convention

Consider two factors and two observed variables with

\Lambda= \begin{pmatrix} 2&0\\ 1&3 \end{pmatrix}, \qquad \Phi=I_2.

The common covariance is

\Lambda\Lambda^{\mathsf T} = \begin{pmatrix} 4&2\\ 2&10 \end{pmatrix}.

Suppose we restrict the loading matrix to be lower triangular and require its diagonal entries to be positive. Among orthogonal rotations of this particular matrix, these conditions select only the original orientation. To see this, write $Q=(q_{ij})$. The first row of $\Lambda Q$ is $(2q_{11},2q_{12})$. Lower triangular form requires $2q_{12}=0$, so $q_{12}=0$. Orthogonality then forces $q_{11}\in\{-1,1\}$, $q_{21}=0$, and $q_{22}\in\{-1,1\}$. Positivity of the first diagonal entry $2q_{11}$ forces $q_{11}=1$; positivity of the second diagonal entry $3q_{22}$ forces $q_{22}=1$. Therefore $Q=I_2$.

This calculation illustrates the role of a triangular loading convention: a zero restriction removes a rotational degree of freedom in this example, and diagonal sign restrictions resolve the remaining sign choices. The calculation does not establish that this restriction is scientifically appropriate, or that it guarantees identification in every factor model. Its success here depends on the stated nonzero diagonal entries and on restricting attention to orthogonal rotations of this loading matrix.

Conventions, Restrictions, and Identification

A scale or orientation constraint can have different interpretations. Setting $\Phi=I_k$ is often a normalization that chooses standardized, uncorrelated factors as the representation. A zero loading may instead encode a substantive claim that a particular variable does not load on a particular factor. A triangular form can be used as an orientation convention, but its zeros are still restrictions on the loading representation. The mathematical role must be stated, rather than inferred from the constraint's appearance.

Even after scale and orientation are addressed, full identification does not follow automatically. The covariance model may admit other parameter values that produce the same $\Sigma$, beyond the transformations considered here. Conversely, signs and permutations can remain as discrete ambiguities: changing the sign of a factor and its loading column together preserves the covariance, and relabelling factors together with their loading columns can do so as well. A positivity convention or an ordering convention can choose among such representations, but those choices do not create information in the observed covariance.

State the factor covariance restriction.
Distinguish $\Phi=I_k$ from merely requiring unit diagonal entries. The former imposes both unit variances and zero factor covariances.

Identify transformations that preserve the restriction.
With $\Phi=I_k$, these include all orthogonal rotations before loading restrictions are applied.

Check what the loading constraints remove.
Verify the constraints algebraically and note any assumptions, such as nonzero anchor loadings or positive diagonal entries.

Separate a chosen representation from identification.
Removing a particular scale or rotation ambiguity does not prove that all observationally equivalent parameter values have been removed.

Common Pitfall

A common error is to say, “The factor variances are set to one, so the factors and loadings are identified.” This conclusion is too strong. If the intended restriction is $\Phi=I_k$, orthogonal rotations still preserve that restriction and the covariance. If only the diagonal of $\Phi$ is fixed to one, factor correlations may remain free, and nonorthogonal transformations can preserve unit diagonal entries. The correction is to specify the entire factor covariance restriction, derive its allowable transformations, and then assess what additional loading restrictions accomplish. A normalization is a coordinate choice; identification is a property of the complete model and its observable implications.

Key takeaway: Factor variance constraints address scale only in the specified parameterization. With identity factor covariance, orthogonal rotations remain; loading restrictions can choose an orientation, but they do not by themselves establish full model identification.

Check Your Understanding

Use the covariance transformation and the distinction between normalization and identification to answer these questions.

For a one-factor model, what changes to $F$, $\Lambda$, and $\Phi$ preserve $\Lambda\Phi\Lambda^{\mathsf T}$ when the factor is rescaled by a nonzero constant?
Why does requiring $\Phi=I_k$ leave orthogonal rotations of the loading matrix allowable?
How does “unit factor variances” differ from “unit variances and zero factor covariances”?
In the triangular example, which condition forces $q_{12}=0$, and which conditions fix the signs of $q_{11}$ and $q_{22}$?
Why does choosing a loading orientation not, by itself, prove full identification of the factor model?