Principal-Factor Extraction from a Reduced Correlation Matrix

From Correlations to a Reduced Matrix

In a common-factor model for standardized variables, each diagonal entry of the correlation matrix is one: it accounts for both common variance and variable-specific variance. Principal-factor extraction starts by estimating the common variance, or communality, for each variable. It then replaces the unit diagonal with those initial estimates and extracts loadings from the resulting reduced correlation matrix.

The eigenvectors describe directions in the variables, while the associated positive eigenvalues determine the scale of the loadings. The calculation is direct once the initial communalities have been chosen. Those estimates matter: changing the diagonal can change both the eigenvalues and eigenvectors.

Definition: Let \(R\) be a \(p \times p\) correlation matrix and let \(h_i^2\) be an initial communality estimate for variable \(i\). The reduced correlation matrix is \(R^*\), with entries \(R^*_{ij}=R_{ij}\) for \(i\ne j\) and \(R^*_{ii}=h_i^2\). Equivalently, \(R^*=R-\operatorname{diag}(1-h_1^2,\ldots,1-h_p^2)\).

A common initial choice, when \(R\) is positive definite, is the squared multiple correlation (SMC) of each variable with all the others. For standardized variables, this is the coefficient of determination from the linear regression of variable \(i\) on the remaining variables. Other initial choices are possible; the extraction below treats the selected estimates as fixed inputs.

Definition: Suppose the leading \(m\) eigenvalues of \(R^*\), in algebraically decreasing order, are positive and have orthonormal eigenvectors \(v_1,\ldots,v_m\). Write \(\Lambda_m=\operatorname{diag}(\lambda_1,\ldots,\lambda_m)\) and \(V_m=[v_1,\ldots,v_m]\). The principal-factor loadings from these eigenpairs are \(L=V_m\Lambda_m^{1/2}\). Thus the loading for variable \(i\) on factor \(j\) is \(L_{ij}=\sqrt{\lambda_j}(v_j)_i\).

The choice of eigenvector sign is arbitrary: replacing \(v_j\) by \(-v_j\) reverses the signs of that factor's loadings without changing their product \(LL^\mathsf{T}\). If an eigenvalue is repeated, the eigenvectors within its eigenspace are not uniquely determined, but an orthonormal basis for that eigenspace still gives the same spectral component when the whole repeated eigenspace is retained.

Worked Calculations

Worked Example: One Factor from an Equicorrelation Matrix

Suppose three standardized variables have the correlation matrix below, and use the stated initial communality estimate of \(0.60\) for each variable. Extract one factor.

R= \begin{pmatrix} 1&0.50&0.50\\ 0.50&1&0.50\\ 0.50&0.50&1 \end{pmatrix}, \qquad R^*= \begin{pmatrix} 0.60&0.50&0.50\\ 0.50&0.60&0.50\\ 0.50&0.50&0.60 \end{pmatrix}.

Each row of \(R^*\) sums to \(1.60\), so \(v_1=(1,1,1)^\mathsf{T}/\sqrt{3}\) is an eigenvector with eigenvalue \(\lambda_1=1.60\). Any vector whose entries sum to zero is an eigenvector with eigenvalue \(0.60-0.50=0.10\), so the other two eigenvalues are \(0.10\). The leading eigenpair is therefore the one used for one-factor extraction.

The loading vector is

L=\sqrt{1.60}\frac{1}{\sqrt{3}} \begin{pmatrix}1\\1\\1\end{pmatrix} = \begin{pmatrix}\sqrt{1.60/3}\\\sqrt{1.60/3}\\\sqrt{1.60/3}\end{pmatrix} \approx \begin{pmatrix}0.730\\0.730\\0.730\end{pmatrix}.

The retained component \(LL^\mathsf{T}\) has every entry equal to \(1.60/3\approx0.533\). In particular, its off-diagonal entries approximate the observed correlations of \(0.50\); they need not reproduce them exactly. Its diagonal is also the retained common variance for each variable, which here is approximately \(0.533\), not the initial estimate \(0.60\).

Worked Example: SMC Initial Communalities and a Negative Eigenvalue

Consider three standardized variables with observed correlation matrix

R= \begin{pmatrix} 1&0.40&0.40\\ 0.40&1&0.40\\ 0.40&0.40&1 \end{pmatrix}.

The common diagonal of the inverse of an equicorrelation matrix with three variables and off-diagonal correlation \(r\) is \((1+r)/((1-r)(1+2r))\). With \(r=0.40\), this is \(1.40/(0.60\times1.80)=35/27\). The SMC of each variable is therefore \(1-1/(35/27)=8/35\). Replacing the diagonal of \(R\) gives

R^*= \begin{pmatrix} 8/35&14/35&14/35\\ 14/35&8/35&14/35\\ 14/35&14/35&8/35 \end{pmatrix}.

The all-ones direction has eigenvalue \(8/35+2(14/35)=36/35\). The two-dimensional subspace of vectors whose entries sum to zero has eigenvalue \(8/35-14/35=-6/35\). Thus there is one positive leading eigenvalue. For a one-factor extraction, choose \(v_1=(1,1,1)^\mathsf{T}/\sqrt{3}\), giving

L=\sqrt{36/35}\frac{1}{\sqrt{3}} \begin{pmatrix}1\\1\\1\end{pmatrix} = \begin{pmatrix}\sqrt{12/35}\\\sqrt{12/35}\\\sqrt{12/35}\end{pmatrix} \approx \begin{pmatrix}0.586\\0.586\\0.586\end{pmatrix}.

The negative eigenvalues cannot be square-rooted to produce real loading columns. They are not included in this one-factor loading matrix. This example also shows that a reduced matrix need not be positive semidefinite, even when the original correlation matrix is positive definite.

Worked Example: Selecting the Leading Eigenpairs in a Block Matrix

Suppose the observed correlations and initial communalities give the following reduced matrix. Extract two factors.

R^*= \begin{pmatrix} 0.80&0.60&0\\ 0.60&0.80&0\\ 0&0&0.30 \end{pmatrix}.

For the first two variables, the vectors \((1,1,0)^\mathsf{T}/\sqrt{2}\) and \((1,-1,0)^\mathsf{T}/\sqrt{2}\) have eigenvalues \(0.80+0.60=1.40\) and \(0.80-0.60=0.20\), respectively. The vector \((0,0,1)^\mathsf{T}\) has eigenvalue \(0.30\). In decreasing order, the leading two eigenpairs are \((1.40,(1,1,0)^\mathsf{T}/\sqrt{2})\) and \((0.30,(0,0,1)^\mathsf{T})\).

Multiplying each unit eigenvector by the square root of its eigenvalue gives

L= \begin{pmatrix} \sqrt{1.40}/\sqrt{2}&0\\ \sqrt{1.40}/\sqrt{2}&0\\ 0&\sqrt{0.30} \end{pmatrix} \approx \begin{pmatrix} 0.837&0\\ 0.837&0\\ 0&0.548 \end{pmatrix}.

The first retained component contributes a covariance of \(1.40/2=0.70\) between variables 1 and 2. The second contributes \(0.30\) to the variance of variable 3. The smaller eigenvalue \(0.20\) is not retained because the requested extraction has two factors and the eigenvalues are ranked before selection.

What the Eigenpair Calculation Guarantees

Theorem: Let \(R^*\) be real and symmetric, with orthonormal eigenvectors \(v_1,\ldots,v_p\) and corresponding eigenvalues \(\lambda_1,\ldots,\lambda_p\). If the selected \(m\) eigenvalues are positive, the loadings \(L=V_m\Lambda_m^{1/2}\) satisfy \(LL^\mathsf{T}=\sum_{j=1}^m\lambda_jv_jv_j^\mathsf{T}\). The residual \(E=R^*-LL^\mathsf{T}\) is \(\sum_{j=m+1}^p\lambda_jv_jv_j^\mathsf{T}\) when the eigenpairs are ordered with the selected ones first. In particular, \(V_m^\mathsf{T}E=0\) and \(\lVert E\rVert_F^2=\sum_{j=m+1}^p\lambda_j^2\).

Proof. By the spectral decomposition of a real symmetric matrix, \(R^*=\sum_{j=1}^p\lambda_jv_jv_j^\mathsf{T}\). Since \(L=V_m\Lambda_m^{1/2}\), multiplication gives \(LL^\mathsf{T}=V_m\Lambda_mV_m^\mathsf{T}=\sum_{j=1}^m\lambda_jv_jv_j^\mathsf{T}\). Subtracting this expression from the full decomposition gives the stated residual. For \(k\le m\) and \(j>m\), orthonormality gives \(v_k^\mathsf{T}v_j=0\), so \(V_m^\mathsf{T}E=0\). Finally, the matrices \(v_jv_j^\mathsf{T}\) are orthonormal under the Frobenius inner product because

\operatorname{tr}\!\left((v_jv_j^\mathsf{T})^\mathsf{T}v_kv_k^\mathsf{T}\right) =(v_j^\mathsf{T}v_k)^2 = \begin{cases} 1,&j=k,\\ 0,&j\ne k. \end{cases}

Consequently the squared Frobenius norm of the residual sum is the sum of the squared coefficients, \(\lVert E\rVert_F^2=\sum_{j=m+1}^p\lambda_j^2\). \(\square\)

This result describes exactly which spectral components the extraction retains. It does not say that the initial communalities are correct, that the resulting factor model fits every off-diagonal correlation, or that a reduced matrix with negative eigenvalues yields real loadings for every eigenpair.

Why the SMC Formula Holds

For completeness, the SMC initial estimate can be derived from a regression calculation. Reorder the variables so the variable of interest is first, and partition the correlation matrix as

R= \begin{pmatrix} 1&r^\mathsf{T}\\ r&R_{-i} \end{pmatrix},

where \(R_{-i}\) is the correlation matrix of the other variables and \(r\) is the vector of their correlations with variable \(i\). Assume \(R_{-i}\) is invertible. The least-squares regression coefficients are \(R_{-i}^{-1}r\), so the squared multiple correlation is \(h_i^2=r^\mathsf{T}R_{-i}^{-1}r\). The block inverse formula gives

(R^{-1})_{ii}=\frac{1}{1-r^\mathsf{T}R_{-i}^{-1}r} =\frac{1}{1-h_i^2}.

Rearranging yields \(h_i^2=1-1/(R^{-1})_{ii}\). The formula requires the relevant inverses to exist; in particular, it applies when \(R\) is positive definite. If the correlation matrix is singular, this inverse-based SMC formula is not directly available.

Interpretation and Common Pitfalls

The reduced matrix is a working target for the common part of the correlations. Its off-diagonal entries remain the observed correlations, while its diagonal contains estimated common variances rather than total variances. The leading eigenpairs identify the strongest retained spectral directions in that target. The resulting loadings are therefore dependent on both the chosen initial communalities and the number of factors retained.

Using eigenpairs of the unreduced matrix. The original correlation matrix has ones on its diagonal. Principal-factor extraction instead decomposes \(R^*\), whose diagonal has been replaced by the initial communalities.
Taking square roots of a negative eigenvalue. A negative eigenvalue has no real square root, so it cannot yield a real loading column by this construction. Inspect the eigenvalues before forming loadings.
Assuming the retained loadings reproduce the input exactly. The product \(LL^\mathsf{T}\) contains only the selected spectral components. Its off-diagonal entries can differ from the original correlations, and its diagonal can differ from the initial communalities.
Treating initial communalities as final estimates. They set the diagonal for this extraction. They are not guaranteed to equal the diagonal of the retained common component.

Key takeaway: Replace the unit diagonal of \(R\) with chosen initial communalities, find the leading positive eigenpairs of \(R^*\), and scale each selected unit eigenvector by the square root of its eigenvalue. The resulting loading matrix represents the retained spectral component, not an exact reproduction of the original correlations.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

What entries change when constructing \(R^*\) from \(R\), and what do the replacement diagonal entries represent?
For a selected eigenpair \((\lambda_j,v_j)\) with \(\lambda_j>0\), how is its loading column calculated?
Why can a positive definite correlation matrix produce a reduced correlation matrix with a negative eigenvalue?
What assumption is needed to use \(1-1/(R^{-1})_{ii}\) as the SMC initial communality?
If fewer eigenpairs are retained, why can \(LL^\mathsf{T}\) fail to reproduce the off-diagonal correlations exactly?