Tutorials › Multivariate Statistics (Graduate) › MINRES Factor Extraction

Home  /  Tutorials  /  Factor Analysis and Identification

Factor Analysis and Identification · Tutorial 543 of 975

MINRES Factor Extraction

Learn how MINRES fits the correlations a factor model reproduces, how to read its residual equations, and how to check an extracted solution.

Intermediate 10 min read

What You'll Learn

  • Define the MINRES objective for a specified number of factors
  • Express its residual matrix and relate it to the full correlation matrix
  • Derive the stationarity equations for unconstrained loadings
  • Solve exact one- and two-factor off-diagonal fits
  • Check communalities, uniquenesses, and positive definiteness
  • Distinguish an exact fit from a merely stationary MINRES solution

Why MINRES Fits Off-Diagonal Correlations

In the common-factor model, observed variables share covariance through a smaller set of common factors; variable-specific variation is represented separately. For a correlation matrix, the diagonal entries are all one, while the off-diagonal entries describe relationships between distinct variables. MINRES, short for minimum residual, chooses loadings to fit those off-diagonal correlations by least squares. It does not directly minimize discrepancies in the diagonal entries.

This distinction matters because the diagonal of a correlation matrix can be matched by assigning each variable a suitable uniqueness once its communality is specified. The informative fitting task for MINRES is therefore the pattern of correlations among different variables. The criterion is a fitting objective, not a likelihood; Gaussian likelihood-based fitting is a separate approach.

Definition: Let \(R=(r_{ij})\) be a \(p\times p\) correlation matrix and fix a number of factors \(q\). For a loading matrix \(L\in\mathbb{R}^{p\times q}\), define the reproduced common covariance matrix \(LL'\), and define the off-diagonal residuals by \[ e_{ij}(L)=r_{ij}-l_i'l_j,\qquad i\ne j, \] where \(l_i'\) is row \(i\) of \(L\). The MINRES objective is \[ F(L)=\sum_{i<j}\left(r_{ij}-l_i'l_j\right)^2. \] A MINRES solution is a global minimizer of \(F\) over the chosen parameter space. Some procedures seek a local minimum instead, so the optimization method and constraints should be reported when relevant.

The criterion counts each unordered pair once. Equivalently, let \(E(L)\) be the symmetric matrix with entries \(e_{ij}(L)\) off the diagonal and zeros on the diagonal. Then \[ F(L)=\frac{1}{2}\|E(L)\|_F^2, \] because the squared Frobenius norm counts each symmetric off-diagonal residual twice. This residual matrix is not the entire difference \(R-LL'\): its diagonal is deliberately set to zero.

Residual Structure and Stationarity

Write \(h_i^2=l_i'l_i\) for the communality of variable \(i\). In a correlation factor model with diagonal uniqueness matrix \(\Psi\), the model-implied correlation matrix is \(LL'+\Psi\). To match its diagonal to one, set \(\psi_i=1-h_i^2\). This gives the MINRES-implied matrix \[ \widehat R=LL'+\operatorname{diag}(1-h_1^2,\ldots,1-h_p^2). \] Its diagonal is one, and its off-diagonal entries are \(l_i'l_j\). Thus \(R-\widehat R=E(L)\), with zero diagonal, and the MINRES objective is precisely half the squared Frobenius norm of this model residual.

For a proper common-factor interpretation with nonnegative uniquenesses, require \(h_i^2\leq 1\) for every \(i\); strict positivity of uniqueness requires \(h_i^2<1\). If an unconstrained numerical MINRES fit gives \(h_i^2>1\), its off-diagonal fit may still be defined, but it does not yield nonnegative uniqueness for the stated correlation model. This is a Heywood-type boundary issue, not a reason to silently alter the objective.

Theorem: At any unconstrained local minimum of \(F(L)\), the zero-diagonal residual matrix satisfies \[ E(L)L=0. \] In row form, for every variable \(i\), \[ \sum_{j\ne i} e_{ij}(L)l_j=0. \] These are necessary stationarity conditions; they do not by themselves guarantee a minimum or a global solution.

Proof. Let \(l_i\) denote the column vector corresponding to row \(i\) of \(L\). For a perturbation \(d l_i\), only residuals involving variable \(i\) change. For each \(j\ne i\), \[ d e_{ij}=-l_j'd l_i. \] The objective can be written \(F=\sum_{a<b}e_{ab}^2\), so its differential with respect to \(l_i\) is \[ dF=2\sum_{j\ne i}e_{ij}\,d e_{ij} =-2\sum_{j\ne i}e_{ij}l_j'd l_i =-2\left(\sum_{j\ne i}e_{ij}l_j\right)'d l_i. \] At an unconstrained local minimum the gradient with respect to each free row \(l_i\) is zero. Hence \(\sum_{j\ne i}e_{ij}l_j=0\) for every \(i\). The \(i\)th row of \(E(L)L\) is exactly this sum, since \(E\) has zero diagonal. Therefore \(E(L)L=0\). \(\square\)

The equations express a balance: at a stationary solution, each variable's vector of residual correlations has zero weighted sum against the other variables' loading vectors. But the objective is generally nonconvex in \(L\), since it contains products of loadings. Stationary points can be saddles or local minima, and changing starting values or constraints can change a numerical result. Also, replacing \(L\) by \(LQ\) for an orthogonal \(q\times q\) matrix \(Q\) leaves \(LL'\), and therefore the objective, unchanged.

Worked Example: An Exact One-Factor Fit for Three Variables

Suppose \[ R=\begin{pmatrix} 1&0.30&0.20\\ 0.30&1&0.40\\ 0.20&0.40&1 \end{pmatrix} \] and fit one factor. For scalar loadings \(l_1,l_2,l_3\), an exact off-diagonal fit must satisfy \(l_1l_2=0.30\), \(l_1l_3=0.20\), and \(l_2l_3=0.40\). Multiplying the first two equations and dividing by the third gives \[ l_1^2=\frac{(0.30)(0.20)}{0.40}=0.15. \] Likewise, \[ l_2^2=\frac{(0.30)(0.40)}{0.20}=0.60,\qquad l_3^2=\frac{(0.20)(0.40)}{0.30}=\frac{4}{15}\approx0.2667. \] Choose the all-positive solution \(l_1=\sqrt{0.15}\), \(l_2=\sqrt{0.60}\), \(l_3=\sqrt{4/15}\). Its products are \(0.30\), \(0.20\), and \(0.40\), respectively, so every off-diagonal residual is zero and \(F(L)=0\), the smallest possible objective.

The communalities are \(0.15\), \(0.60\), and \(4/15\); the corresponding uniquenesses are \(0.85\), \(0.40\), and \(11/15\). All are positive. The input is also a valid positive-definite correlation matrix: its leading principal minors are \(1\), \(1-0.30^2=0.91\), and \[ \det(R)=1+2(0.30)(0.20)(0.40)-0.30^2-0.20^2-0.40^2=0.758, \] all positive. Thus the exact off-diagonal fit is compatible with positive uniquenesses and a valid input matrix.

Worked Two-Factor Extraction

For a fixed \(q\), a zero MINRES objective is possible exactly when the off-diagonal entries of \(R\) can be represented as pairwise inner products \(l_i'l_j\). The diagonal of \(LL'\) need not equal the diagonal of \(R\), because the uniquenesses supply the difference. The next example gives an exact two-factor fit and also checks the matrix's eigenvalues.

Worked Example: A Two-Factor Fit for Four Variables

Consider the correlation matrix with within-pair correlations \(0.4\) and cross-pair correlations \(0.1\): \[ R=\begin{pmatrix} 1&0.4&0.1&0.1\\ 0.4&1&0.1&0.1\\ 0.1&0.1&1&0.4\\ 0.1&0.1&0.4&1 \end{pmatrix}. \] Define two loading vectors \[ u=\left(\sqrt{0.4},0\right),\qquad v=\left(\frac{0.1}{\sqrt{0.4}},\sqrt{0.4-\frac{0.01}{0.4}}\right) =\left(\sqrt{0.025},\sqrt{0.375}\right). \] Take rows 1 and 2 of \(L\) to be \(u\), and rows 3 and 4 to be \(v\).

The within-pair products are \(u'u=0.4\) and \(v'v=0.025+0.375=0.4\). A cross-pair product is \[ u'v=\sqrt{0.4}\frac{0.1}{\sqrt{0.4}}+0\cdot\sqrt{0.375}=0.1. \] These calculations verify every off-diagonal entry of \(LL'\). Therefore all residuals vanish and \(F(L)=0\). Each communality is \(0.4\), so setting each uniqueness to \(0.6\) makes the diagonal of \(LL'+\Psi\) equal to one.

To check positive definiteness independently, vectors contrasting the two variables within either pair are eigenvectors with eigenvalue \(1-0.4=0.6\), giving two eigenvalues of \(0.6\). On the subspace of vectors constant within each pair, \(R\) reduces to \[ \begin{pmatrix}1.4&0.2\\0.2&1.4\end{pmatrix}, \] whose eigenvalues are \(1.4+0.2=1.6\) and \(1.4-0.2=1.2\). Thus the four eigenvalues are \(0.6,0.6,1.2,1.6\), all positive; their sum is \(4\), as required by the trace.

Using the Objective When Residuals Remain

An extraction need not fit every correlation exactly. The objective gives a direct way to summarize the remaining discrepancy, while the residual matrix shows where the discrepancy occurs. The following calculation illustrates how to evaluate a candidate rather than treating a displayed loading pattern as automatically optimal.

Worked Example: Calculating a Nonzero MINRES Objective

Let the observed off-diagonal correlations be \(r_{12}=0.40\), \(r_{13}=0.10\), and \(r_{23}=0.30\). Evaluate the one-factor candidate \(L=(0.50,0.50,0.50)'\). Every reproduced off-diagonal correlation is \(0.50(0.50)=0.25\), so the residuals are \[ e_{12}=0.40-0.25=0.15,\quad e_{13}=0.10-0.25=-0.15,\quad e_{23}=0.30-0.25=0.05. \] Consequently, \[ F(L)=0.15^2+(-0.15)^2+0.05^2 =0.0225+0.0225+0.0025=0.0475. \] The residual matrix has zero diagonal and the three residuals mirrored across the diagonal. Its squared Frobenius norm is \(2(0.0475)=0.095\), confirming \(F=\tfrac12\|E\|_F^2\).

This is an objective value for the stated candidate, not evidence that the candidate is a MINRES solution. For example, the stationarity condition for the first loading would require \(e_{12}l_2+e_{13}l_3=0\), but here it is \((0.15)(0.50)+(-0.15)(0.50)=0\). For the second loading it is \(e_{12}l_1+e_{23}l_3=(0.15)(0.50)+(0.05)(0.50)=0.10\), which is not zero. Thus this candidate fails the unconstrained stationarity equations and cannot be an unconstrained local minimum.

Interpretation, Constraints, and a Common Pitfall

MINRES is useful when the intended target is the correlation pattern among distinct variables, with no likelihood assumption built into the fitting criterion. Its residuals identify which pairs are under- or over-reproduced. Reporting only a single objective value can conceal that pattern; reporting the residual matrix, loadings, communalities, and uniquenesses makes the fit easier to assess. As the examples show, a zero objective establishes an exact off-diagonal representation, while nonnegative uniquenesses must still be checked separately.

A common mistake is to minimize every entry of \(R-LL'\) and call the result MINRES. That includes diagonal discrepancies, even though standard MINRES excludes them and matches the diagonal through uniquenesses. Another mistake is to treat \(E(L)L=0\) as a sufficient condition for an optimum: it is only a necessary condition for an unconstrained local minimum. If uniqueness constraints are imposed, a solution on a boundary instead requires the appropriate constrained optimality conditions; the unconstrained equations alone do not settle it.

Key takeaway: MINRES minimizes squared residuals for \(i\ne j\), with the diagonal handled by uniquenesses. Its zero-diagonal residual matrix obeys \(E(L)L=0\) at an unconstrained stationary point, but this condition is not a guarantee of global optimality or admissible uniquenesses.

Check Your Understanding

Use the definitions and calculations above to check your understanding.

  1. Why does the Frobenius-norm expression for the MINRES objective require a factor of one half?
  2. For a one-factor fit, which three loading products must match the off-diagonal correlations in a three-variable matrix?
  3. What does \(E(L)L=0\) state in row-wise form, and why is it not sufficient to establish a global minimum?
  4. If a fitted communality is \(1.08\) in a correlation model, what is the implied uniqueness, and what interpretation problem follows?
  5. In the four-variable worked example, explain how the eigenvalues on the pair-constant subspace are obtained.

Previous

Next

Back to Tutorials
Clinical Biostats