Recovering One-Factor Loadings from Covariances

From Covariances to Loadings

A one-factor model describes covariation among several variables through a single shared source. Once the factor variance has been fixed to set its scale, the off-diagonal entries of the population covariance matrix encode products of pairs of loadings. With at least three indicators and nonzero loadings, those products can determine each loading’s magnitude. They cannot determine whether all loadings have been multiplied by \(-1\).

This distinction matters in practice: loadings and uniquenesses describe different parts of the covariance structure, and sign choices affect how loadings are reported without changing that structure. The calculations below concern the population covariance matrix, not the sampling variability or estimation of a covariance matrix from data.

Definition: In the one-factor covariance model for \(p\) variables, write \(X=\lambda F+\varepsilon\), where \(X\) is a \(p\)-vector, \(\lambda=(\lambda_1,\ldots,\lambda_p)^\mathsf{T}\) is the loading vector, \(F\) is a scalar factor with \(\operatorname{Var}(F)=1\), and \(\varepsilon\) is a \(p\)-vector of specific errors. Assume \(\operatorname{Cov}(F,\varepsilon)=0\) and \(\operatorname{Cov}(\varepsilon)=\Psi=\operatorname{diag}(\psi_1,\ldots,\psi_p)\), with \(\psi_i\geq 0\). Then the population covariance matrix is \(\Sigma=\lambda\lambda^\mathsf{T}+\Psi\). The quantities \(\psi_i\) are the uniquenesses.

The unit-variance convention for \(F\) fixes the scale of the factor. Under this convention, the \(i\)th diagonal entry and any off-diagonal entry are

\sigma_{ii}=\lambda_i^2+\psi_i, \qquad \sigma_{ij}=\lambda_i\lambda_j\quad(i\ne j).

The diagonal combines a common part, \(\lambda_i^2\), and a specific part, \(\psi_i\). By contrast, an off-diagonal covariance has no specific-error contribution under the diagonal-\(\Psi\) assumption. It records the product of two loadings.

Recovering Magnitudes and Uniquenesses

For three distinct indices \(i,j,k\), multiply the covariances involving \(i\), then divide by the covariance between \(j\) and \(k\). If all three loadings are nonzero, this gives

\frac{\sigma_{ij}\sigma_{ik}}{\sigma_{jk}} = \frac{(\lambda_i\lambda_j)(\lambda_i\lambda_k)} {\lambda_j\lambda_k} =\lambda_i^2.

Thus three off-diagonal covariances recover a squared loading. The diagonal then gives the corresponding uniqueness. The following result states the conditions under which this argument recovers all loadings’ magnitudes from \(\Sigma\).

Theorem: Suppose \(p\geq 3\) and a population covariance matrix has a one-factor representation \(\Sigma=\lambda\lambda^\mathsf{T}+\Psi\) with every \(\lambda_i\ne 0\). For any two distinct indices \(j,k\) different from \(i\), \(\lambda_i^2=\sigma_{ij}\sigma_{ik}/\sigma_{jk}\) and \(\psi_i=\sigma_{ii}-\lambda_i^2\). The loading magnitudes and uniquenesses are therefore determined by \(\Sigma\). The loading vector itself is determined only up to the two choices \(\lambda\) and \(-\lambda\).

Proof. The model gives \(\sigma_{ij}=\lambda_i\lambda_j\), \(\sigma_{ik}=\lambda_i\lambda_k\), and \(\sigma_{jk}=\lambda_j\lambda_k\). Since the loadings are nonzero, \(\sigma_{jk}\ne 0\), and division yields

\frac{\sigma_{ij}\sigma_{ik}}{\sigma_{jk}} = \frac{\lambda_i^2\lambda_j\lambda_k}{\lambda_j\lambda_k} =\lambda_i^2.

The diagonal equation then gives \(\psi_i=\sigma_{ii}-\lambda_i^2\). To check the sign ambiguity, suppose another loading vector \(\lambda'\) gives the same off-diagonal covariances. Because every \(\lambda_i\ne0\), define \(r_i=\lambda_i'/\lambda_i\). For every distinct pair, equality of the off-diagonal products implies \(r_ir_j=1\). Choose three distinct indices \(i,j,k\). From \(r_ir_j=1\) and \(r_ir_k=1\), we obtain \(r_j=r_k\). Then \(r_jr_k=1\) gives \(r_j^2=1\), so \(r_j=r_k\) is either \(1\) or \(-1\). The pair equations force every other \(r_i\) to have this same value. Hence \(\lambda'=\lambda\) or \(\lambda'=-\lambda\). Both choices give the same \(\lambda\lambda^\mathsf{T}\), and the diagonal equations give the same uniquenesses. \(\square\)

The formula gives a direct way to recover magnitudes, but a covariance matrix supplied without a known factor representation should also be checked for compatibility with the model. For example, each computed square must be nonnegative, and each resulting uniqueness must be nonnegative. With more than three variables, different choices of \(j,k\) for the same \(i\) must give the same square. These are checks on a proposed representation, not additional parameters to estimate.

Worked Example: Three Variables and a Global Sign Choice

Suppose the covariance matrix is

\Sigma= \begin{pmatrix} 1&-0.30&0.48\\ -0.30&1&-0.40\\ 0.48&-0.40&1 \end{pmatrix}.

All three off-diagonal entries are nonzero, so use the three-indicator formulas. For the first variable,

\lambda_1^2 =\frac{\sigma_{12}\sigma_{13}}{\sigma_{23}} =\frac{(-0.30)(0.48)}{-0.40} =\frac{-0.144}{-0.40} =0.36.

For the second and third variables, respectively,

\lambda_2^2 =\frac{\sigma_{12}\sigma_{23}}{\sigma_{13}} =\frac{(-0.30)(-0.40)}{0.48} =\frac{0.12}{0.48} =0.25, \qquad \lambda_3^2 =\frac{\sigma_{13}\sigma_{23}}{\sigma_{12}} =\frac{(0.48)(-0.40)}{-0.30} =\frac{-0.192}{-0.30} =0.64.

The magnitudes are therefore \(0.6,0.5,0.8\). Choose \(\lambda_1=0.6\) as a sign convention. Since \(\sigma_{12}=\lambda_1\lambda_2=-0.30\), this choice requires \(\lambda_2=-0.5\). Since \(\sigma_{13}=\lambda_1\lambda_3=0.48\), it requires \(\lambda_3=0.8\). The recovered vector is \((0.6,-0.5,0.8)^\mathsf{T}\); its negative gives exactly the same covariance matrix.

Each diagonal is one, so the uniquenesses are \(\psi_1=1-0.36=0.64\), \(\psi_2=1-0.25=0.75\), and \(\psi_3=1-0.64=0.36\). Checking the products gives \((0.6)(-0.5)=-0.30\), \((0.6)(0.8)=0.48\), and \((-0.5)(0.8)=-0.40\), as required.

Worked Example: Checking Recovery With Four Variables

Consider a four-variable covariance matrix whose diagonal entries are all one and whose off-diagonal entries are \(\sigma_{12}=0.20\), \(\sigma_{13}=-0.30\), \(\sigma_{14}=0.15\), \(\sigma_{23}=-0.24\), \(\sigma_{24}=0.12\), and \(\sigma_{34}=-0.18\).

For variable 1, using variables 2 and 3 gives \(\lambda_1^2=(0.20)(-0.30)/(-0.24)=0.25\). Using variables 2 and 4 gives the same result: \(\lambda_1^2=(0.20)(0.15)/(0.12)=0.25\). This agreement is an important consistency check.

For variable 2, use variables 1 and 3: \(\lambda_2^2=(0.20)(-0.24)/(-0.30)=0.16\). For variable 3, use variables 1 and 2: \(\lambda_3^2=(-0.30)(-0.24)/(0.20)=0.36\). For variable 4, use variables 1 and 2: \(\lambda_4^2=(0.15)(0.12)/(0.20)=0.09\).

Choose \(\lambda_1=0.5\). The covariances with variable 1 then imply \(\lambda_2=0.20/0.5=0.4\), \(\lambda_3=-0.30/0.5=-0.6\), and \(\lambda_4=0.15/0.5=0.3\). Checking the remaining products gives \((0.4)(-0.6)=-0.24\), \((0.4)(0.3)=0.12\), and \((-0.6)(0.3)=-0.18\). With diagonal entries equal to one, the uniquenesses are \(0.75,0.84,0.64,0.91\), respectively. They are all nonnegative, and the global sign reversal gives the same representation.

Why Two Indicators Are Different

With only two variables, there is just one off-diagonal equation, \(\sigma_{12}=\lambda_1\lambda_2\). A single product does not generally determine two loading magnitudes. The diagonal equations do not resolve this if the uniquenesses are also unknown: changing a loading magnitude can be offset by changing its uniqueness.

Worked Example: One Product, Multiple Loading Pairs

Suppose the two-variable covariance matrix has diagonal entries one and off-diagonal entry \(0.24\):

\Sigma= \begin{pmatrix} 1&0.24\\ 0.24&1 \end{pmatrix}.

One valid loading pair is \((0.4,0.6)\), since \((0.4)(0.6)=0.24\). Its uniquenesses are \(1-0.4^2=0.84\) and \(1-0.6^2=0.64\). Another valid pair is \((0.3,0.8)\), since \((0.3)(0.8)=0.24\); its uniquenesses are \(1-0.3^2=0.91\) and \(1-0.8^2=0.36\). Both have nonnegative uniquenesses and reproduce every entry of \(\Sigma\). Thus, even apart from the global sign ambiguity, the two loading magnitudes are not determined by this covariance matrix.

Common Pitfalls and Model Checks

Reporting a unique loading vector. The reasoning “the covariances determine the loadings, so their signs are fixed” is incorrect. Covariances determine pairwise products, and \(\lambda\) and \(-\lambda\) give identical products. Choose a sign convention, such as \(\lambda_1>0\), to report one representative.
Using only the diagonal to find a loading. Since \(\sigma_{ii}=\lambda_i^2+\psi_i\), the diagonal alone does not separate common variance from uniqueness. Recover \(\lambda_i^2\) from off-diagonal covariances first, then subtract it from the diagonal.
Dividing when the denominator is zero. The recovery formula requires \(\sigma_{jk}\ne0\), which follows from the theorem’s nonzero-loading assumption. If some loadings or covariances are zero, the displayed formulas may not apply; the zero pattern needs separate analysis.
Ignoring admissibility. A computed \(\lambda_i^2<0\) cannot be the square of a real loading. A computed \(\psi_i<0\) violates the stated nonnegative-uniqueness condition. Either result means the proposed matrix does not have the assumed representation under these conditions.

The sign of an individual loading is meaningful only relative to the chosen orientation of the factor: reversing the factor’s direction reverses every loading together. By contrast, the loading magnitudes and uniquenesses determine how much of each variance is assigned to the common factor and to its specific error. This is why recovering them from population covariances is useful, and why the assumptions behind that recovery must be checked rather than taken for granted.

Check Your Understanding

Use the model equations and state any conditions your reasoning requires.

Why does the ratio \(\sigma_{ij}\sigma_{ik}/\sigma_{jk}\) recover a squared loading rather than a signed loading?
If every loading is nonzero and there are at least three variables, what are the two loading vectors that yield the same covariance matrix?
For a recovered squared loading of \(0.49\) and a corresponding diagonal entry of \(1.2\), what is the uniqueness?
Why can two variables with an unknown diagonal uniqueness matrix fail to determine both loading magnitudes?
If two different choices of \(j,k\) give different values for \(\sigma_{ij}\sigma_{ik}/\sigma_{jk}\), what does that tell you about the proposed one-factor representation?

Recovering One-Factor Loadings from Covariances

What You'll Learn