Gram-Schmidt in a Matrix Space

Inner Products for Matrices

Gram–Schmidt is not restricted to vectors written as coordinate columns. Matrices of a fixed size can themselves be treated as vectors: their entries supply the coordinates, and an inner product measures their lengths and angles. This makes the method useful whenever the objects being compared are matrices, such as when building orthonormal lists in a matrix space.

Throughout, the scalar field is \(\mathbb{R}\). For fixed positive integers \(m\) and \(n\), let \(\mathbb{R}^{m\times n}\) be the vector space of \(m\times n\) real matrices. If \(\mathbf{A}=(a_{ij})\) and \(\mathbf{B}=(b_{ij})\) belong to this space, their trace inner product is the sum of the products of corresponding entries. Equivalently, it is the trace of \(\mathbf{A}^{\top}\mathbf{B}\), which is an \(n\times n\) square matrix.

This is also called the Frobenius inner product. The entrywise formula is often the quickest way to calculate with it: pair entries in the same position, multiply, and add. In particular, no matrix multiplication is needed to find an inner product, even though the trace formula gives an equivalent definition.

Why this is an inner product. The trace formula has the required properties because its entrywise expression is an ordinary dot product of the \(mn\) entries. More explicitly, for real scalars \(c,d\) and matrices \(\mathbf{A},\mathbf{B},\mathbf{C}\) of the same size, summing entries gives \(\langle c\mathbf{A}+d\mathbf{B},\mathbf{C}\rangle=c\langle\mathbf{A},\mathbf{C}\rangle+d\langle\mathbf{B},\mathbf{C}\rangle\); the same calculation in the second slot gives linearity there. Also, \(\langle\mathbf{A},\mathbf{B}\rangle=\langle\mathbf{B},\mathbf{A}\rangle\), since real multiplication is commutative. Finally, \(\langle\mathbf{A},\mathbf{A}\rangle=\sum_{i,j}a_{ij}^2\geq0\), and this sum is zero exactly when every entry is zero, that is, exactly when \(\mathbf{A}=\mathbf{O}\).

Projections and Residuals in a Matrix Space

The first Gram–Schmidt step uses a matrix already known to be nonzero. If \(\mathbf{U}\ne\mathbf{O}\), the coefficient needed to remove from \(\mathbf{V}\) its component along \(\mathbf{U}\) is \(\langle\mathbf{V},\mathbf{U}\rangle/\langle\mathbf{U},\mathbf{U}\rangle\). The denominator matters when \(\mathbf{U}\) is not a unit matrix. If instead \(\mathbf{Q}\) has norm \(1\), the coefficient is simply \(\langle\mathbf{V},\mathbf{Q}\rangle\).

Proof. Since \(\mathbf{U}\ne\mathbf{O}\), positive definiteness gives \(\langle\mathbf{U},\mathbf{U}\rangle\ne0\). By linearity in the first slot and symmetry of the real inner product,

For the unit-matrix form, \(\langle\mathbf{Q},\mathbf{Q}\rangle=1\), so the coefficient in the first formula becomes \(\langle\mathbf{V},\mathbf{Q}\rangle/1\). This proves both statements. \(\square\)

For a list of matrices, the same idea is applied successively: after making the first matrix a unit matrix, subtract the components of the next matrix along the unit matrices already constructed, then normalize the new residual. The examples below calculate those components using the trace inner product, rather than relying on visual impressions of the matrices.

Two Matrices: One Projection and One Normalization

A Three-Matrix Calculation

This example also shows why the order of the input list matters: each new residual is formed by removing components along the unit matrices already obtained. The third matrix’s projection onto \(\mathbf{Q}_2\) happens to be zero, but that coefficient is still calculated rather than assumed.

When More Than One Component Must Be Removed

A Common Coefficient Error

A frequent mistake is to subtract the inner product with a non-unit matrix as though that matrix had norm \(1\). In the first worked example, \(\langle\mathbf{A}_2,\mathbf{A}_1\rangle=2\), but \(\|\mathbf{A}_1\|^2=3\). The incorrect calculation \(\mathbf{A}_2-2\mathbf{A}_1\) gives \(\begin{bmatrix}-1&-2\\1&-1\end{bmatrix}\). Its inner product with \(\mathbf{A}_1\) is \(-1-2+0-1=-4\), so this is not orthogonal to \(\mathbf{A}_1\). The correction is to divide by the squared norm: subtract \((2/3)\mathbf{A}_1\), yielding the residual \(\begin{bmatrix}1/3&-2/3\\1&1/3\end{bmatrix}\), whose inner product with \(\mathbf{A}_1\) is \(1/3-2/3+0+1/3=0\).

The distinction is simple to retain: with a non-unit matrix \(\mathbf{U}\), divide by \(\langle\mathbf{U},\mathbf{U}\rangle\); with a unit matrix \(\mathbf{Q}\), the projection coefficient is just \(\langle\mathbf{V},\mathbf{Q}\rangle\). After every subtraction, check orthogonality by an entrywise inner product, and after every normalization, check that the squared entries sum to \(1\).