Orthogonality Of The Left Singular Vectors

From Right Singular Vectors to Orthogonal Images

Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\). A right singular vector is an eigenvector of \(\mathbf{A}^{\top}\mathbf{A}\); if \(\mathbf{v}_i\) is a unit eigenvector with eigenvalue \(\lambda_i>0\), its singular value is \(\sigma_i=\sqrt{\lambda_i}\). The construction of a corresponding left singular vector is to take the image \(\mathbf{A}\mathbf{v}_i\) and divide by \(\sigma_i\). This tutorial establishes why the resulting vectors are orthonormal when the right singular vectors are orthonormal and the singular values are positive.

The hypotheses matter. The vectors \(\mathbf{v}_1,\dots,\mathbf{v}_r\in\mathbb{R}^n\) are assumed to be orthonormal: \(\mathbf{v}_i^{\top}\mathbf{v}_j=0\) when \(i\ne j\), and \(\mathbf{v}_i^{\top}\mathbf{v}_i=1\). They are eigenvectors satisfying \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_i=\lambda_i\mathbf{v}_i\), with \(\lambda_i>0\). Define \(\sigma_i=\sqrt{\lambda_i}\) and \(\mathbf{u}_i=\mathbf{A}\mathbf{v}_i/\sigma_i\). Since \(\sigma_i>0\), this division is defined. The vectors \(\mathbf{u}_i\) belong to \(\mathbb{R}^m\), because \(\mathbf{A}\) maps \(\mathbb{R}^n\) into \(\mathbb{R}^m\).

To check this definition for the constructed vectors, compute their inner products. Matrix multiplication gives \((\mathbf{A}\mathbf{v}_i)^{\top}(\mathbf{A}\mathbf{v}_j)=\mathbf{v}_i^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_j\). The eigenvector equation then replaces \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_j\) by \(\lambda_j\mathbf{v}_j\). This connects the inner products of the images directly to the assumed orthonormality of the original vectors.

The Inner Product Identity

Proof. Fix indices \(i,j\). By the definition of transpose and matrix multiplication,

If \(i\ne j\), orthonormality gives \(\mathbf{v}_i^{\top}\mathbf{v}_j=0\), so the inner product of the images is \(0\). If \(i=j\), orthonormality gives \(\mathbf{v}_i^{\top}\mathbf{v}_i=1\); the displayed identity becomes \((\mathbf{A}\mathbf{v}_i)^{\top}(\mathbf{A}\mathbf{v}_i)=\lambda_i\). These are both claimed cases. \(\square\)

In particular, the squared length of \(\mathbf{A}\mathbf{v}_i\) is \(\lambda_i\). Its length is therefore \(\sqrt{\lambda_i}=\sigma_i\), using the positive square root because \(\lambda_i>0\). Thus dividing by \(\sigma_i\) makes the image a unit vector. For distinct indices, scaling two orthogonal images by nonzero numbers leaves their inner product zero. Both parts of orthonormality follow.

Proof. For any \(i,j\), the proposition gives \((\mathbf{A}\mathbf{v}_i)^{\top}(\mathbf{A}\mathbf{v}_j)=0\) when \(i\ne j\), and \((\mathbf{A}\mathbf{v}_i)^{\top}(\mathbf{A}\mathbf{v}_i)=\lambda_i\). Since every \(\sigma_i\) is positive,

When \(i\ne j\), the numerator is zero and the denominator is nonzero, so \(\mathbf{u}_i^{\top}\mathbf{u}_j=0\). When \(i=j\), the same expression is \(\lambda_i/\sigma_i^2=1\), since \(\sigma_i^2=\lambda_i\). The constructed vectors are pairwise orthogonal and each has unit length, which is exactly the definition of an orthonormal list. \(\square\)

Worked Checks in Rectangular Matrices

What the Hypotheses Do—and Do Not—Say

There is no requirement that the positive eigenvalues be distinct. The proof works for every pair of indices, including pairs whose eigenvalues agree: for distinct indices, the factor \(\mathbf{v}_i^{\top}\mathbf{v}_j=0\) makes the image inner product zero. What matters is that the right eigenvectors used together are orthonormal. If one starts with eigenvectors that are merely nonzero, the theorem's hypothesis has not been met; the conclusion does not follow from the argument unless their inner products are also handled.

Positivity is essential to this particular construction. If \(\lambda_i=0\), then \(\sigma_i=0\), and the quotient \(\mathbf{A}\mathbf{v}_i/\sigma_i\) is undefined. In fact, the identity in the proposition gives \(\|\mathbf{A}\mathbf{v}_i\|^2=0\), so the image itself is the zero vector and cannot be made into a unit vector by scaling. A left vector associated with a zero singular value must therefore be obtained by a different step; dividing by zero is never a valid way to construct it.

Common mistake: treating orthogonal as orthonormal. A tempting but incorrect calculation is to observe that \((3,0,0)^{\top}\) and \((0,2,0)^{\top}\) have inner product zero and conclude that they are orthonormal. Their lengths are \(3\) and \(2\), not \(1\). The correction is to divide each image by its own positive singular value: \((3,0,0)^{\top}/3=(1,0,0)^{\top}\) and \((0,2,0)^{\top}/2=(0,1,0)^{\top}\). Then both lengths are \(1\), while the inner product remains zero.

The central computation can be retained as a compact test: the Gram matrix converts an inner product of two images into an inner product of their right vectors, multiplied by an eigenvalue. For distinct orthonormal right vectors this gives zero; for a vector paired with itself it gives its eigenvalue, which is the square of the normalizing singular value. This is why the positive-image construction produces orthonormal vectors in the output space.