Rank Factorizations and the Moore–Penrose Inverse

Why Factor Rank-Deficient Matrices?

A matrix with deficient rank cannot be inverted in the ordinary sense: it may collapse some directions to zero, or its output space may have lower dimension than its input space. Yet many calculations still need an inverse-like operation that respects the directions the matrix actually retains. The Moore–Penrose inverse provides that operation. Its construction is especially transparent when we first separate a matrix into full-rank factors.

The singular value decomposition (SVD) gives a direct route: retain the positive singular values, invert them, and leave the zero-singular-value directions out. A rank factorization gives another route and a useful computational formula. This tutorial develops both constructions and works through their dimensions and products carefully.

Rank Factorizations

Let \(A\) be an \(m\times n\) matrix with rank \(r\). A rank factorization expresses \(A\) as the product of an \(m\times r\) matrix with full column rank and an \(r\times n\) matrix with full row rank. The inner dimension is exactly the rank; this is not just any factorization.

One construction starts with a compact SVD. Write \(A=U_r\Sigma_rV_r^T\), where the columns of \(U_r\) and \(V_r\) are orthonormal and \(\Sigma_r\) is the \(r\times r\) diagonal matrix of positive singular values. Then \(B=U_r\Sigma_r\) has full column rank and \(C=V_r^T\) has full row rank, giving \(A=BC\). Alternatively, one may choose \(r\) linearly independent columns of \(A\) as \(B\), then express every column of \(A\) as a linear combination of those columns; the coefficients form \(C\).

Definition from the Compact SVD

For \(r>0\), the compact SVD of \(A\) is \(A=U_r\Sigma_rV_r^T\), with \(\Sigma_r=\operatorname{diag}(\sigma_1,\ldots,\sigma_r)\) and \(\sigma_i>0\). The remaining singular values in a full SVD, if present, are zero. They must not be inverted: a reciprocal of zero does not exist, and those directions are precisely the ones \(A\) annihilates.

The dimensions reverse: \(A\) is \(m\times n\), while \(A^+\) is \(n\times m\). The formula reverses the order of the SVD factors and inverts only the positive singular values. For a full SVD, the same construction can be written by replacing each positive singular value by its reciprocal and each zero singular value by zero.

A Formula from Any Rank Factorization

The compact SVD is not the only way to calculate \(A^+\). If \(A=BC\) is any rank factorization, the Gram matrices \(B^TB\) and \(CC^T\) are invertible: for example, \(x^TB^TBx=\|Bx\|^2>0\) for every nonzero \(x\), because \(B\) has full column rank. The corresponding argument applies to \(CC^T\).

Here is a derivation that also explains why the formula agrees with the SVD definition. The matrices \(B^+\) and \(C^+\) have the indicated forms because \(B\) has full column rank and \(C\) has full row rank. In particular, \(B^+B=I_r\) and \(CC^+=I_r\). For \(X=C^+B^+\), direct multiplication gives

and

Both \(BB^+=B(B^TB)^{-1}B^T\) and \(C^+C=C^T(CC^T)^{-1}C\) are symmetric, as is seen by transposing each expression. Thus \(X\) has the defining SVD inverse’s two key symmetry properties as well as \(AXA=A\). More directly connecting the formulas, take the SVD rank factorization \(B=U_r\Sigma_r,\ C=V_r^T\). Substitution gives \(C^+=V_r\) and \(B^+=\Sigma_r^{-1}U_r^T\), so \(C^+B^+=V_r\Sigma_r^{-1}U_r^T\), exactly the compact-SVD formula.

The rank-factorization expression does not depend on which full-rank factors are chosen. To see this, suppose also \(A=B'C'\) is a rank factorization. Both \(B\) and \(B'\) have columns forming bases for the column space of \(A\); hence \(B'=BT\) for an invertible \(r\times r\) matrix \(T\), and then \(C'=T^{-1}C\). Substitution into the full-column and full-row formulas gives \(B'^+=T^{-1}B^+\) and \(C'^+=C^+T\). Therefore \(C'^+B'^+=C^+B^+\). This verifies that the factor formula gives a single matrix for \(A\), regardless of the rank factorization used. \(\square\)

Checking a Rank-Deficient Calculation

A reliable check is to multiply the proposed inverse by a vector, or to verify matrix products, without confusing the entries or dimensions. Consider the symmetric rank-two matrix \(A=\begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix}\). Let \(q_1=(1,-1,0)^T/\sqrt2\) and \(q_2=(1,1,2)^T/\sqrt6\). These are orthonormal, and direct multiplication gives \(Aq_1=q_1\) and \(Aq_2=3q_2\). Also \(A(1,1,-1)^T=0\). Thus an SVD can use \(q_1,q_2\) as the positive-singular-value left and right vectors, with positive singular values \(1\) and \(3\).

The SVD formula therefore gives \(A^+=q_1q_1^T+\frac13q_2q_2^T\). Expanding each outer product,

For \(b=(2,1,0)^T\), the third coordinate of \(A^+b\) is \((2\cdot1+1\cdot1+0\cdot2)/9=3/9=1/3\). In particular, the two nonzero terms sum to \(3\), not \(2\). The full multiplication is \(A^+b=(2/3,-1/3,1/3)^T\). Checking a coordinate this way is useful: every term in a matrix-vector product must use an entry from the selected row and the corresponding entry of the vector.

One can also check the inverse by multiplying the matrices. Since \(A^+\) has eigenvalues \(1,1/3,0\) along \(q_1,q_2,(1,1,-1)^T\), respectively, \(AA^+\) and \(A^+A\) both act as the identity on the first two directions and as zero on the third. Consequently \(AA^+A=A\), \(A^+AA^+=A^+\), and both \(AA^+\) and \(A^+A\) are symmetric.

Common Pitfalls and Practical Value

• Inverting zero singular values. The inverse formula uses only the positive singular values in the compact SVD. A zero singular value is omitted, not reciprocated.
• Forgetting the dimensions. If \(A\) is \(m\times n\), then \(A^+\) is \(n\times m\). The order of the SVD factors reverses.
• Using arbitrary factors in the rank formula. The displayed formula requires full column rank for \(B\), full row rank for \(C\), and exactly \(r\) inner dimensions. Without these conditions, the Gram-matrix inverses need not exist.
• Rounding or summing a product incorrectly. Compute each entry as a row-column dot product, retaining all terms before simplifying. For instance, \((2\cdot1+1\cdot1)/9=3/9=1/3\), not \(2/9\).

The SVD construction is useful because it identifies exactly which directions are retained and how strongly they are scaled. The rank-factorization formula can be more convenient when a matrix is already expressed through lower-dimensional full-rank factors. Both approaches make the rank deficiency explicit rather than hiding it behind an ordinary inverse that does not exist.