Why Orthogonalize Inside an Eigenspace?
An eigenspace can contain many different eigenvectors for the same eigenvalue. When its dimension is greater than one, a basis chosen by solving the eigenvector equations need not consist of perpendicular vectors. Gram–Schmidt replaces such a basis with an orthonormal one: the new vectors are pairwise orthogonal, have length one, and still belong to the original eigenspace. That last property is essential. A perpendicular vector obtained by a calculation is useful as an eigenvector only if it remains in the eigenspace.
Work over \(\mathbb{R}\), with the dot product \(\mathbf{x}\cdot\mathbf{y}=\mathbf{x}^{\top}\mathbf{y}\) and norm \(\|\mathbf{x}\|=\sqrt{\mathbf{x}\cdot\mathbf{x}}\). Let \(\mathbf{A}\in\mathbb{R}^{n\times n}\) be symmetric, so \(\mathbf{A}^{\top}=\mathbf{A}\), and let \(\lambda\) be one of its real eigenvalues. Its eigenspace is \(E_{\lambda}=\{\mathbf{x}\in\mathbb{R}^n:\mathbf{A}\mathbf{x}=\lambda\mathbf{x}\}\). The zero vector belongs to this set, but is not an eigenvector.
Every vector in \(E_{\lambda}\) is an eigenvector only when it is nonzero. Thus an orthonormal basis of \(E_{\lambda}\) consists of eigenvectors, since each of its vectors has norm one. We assume the eigenspace is finite-dimensional, as it is a subspace of \(\mathbb{R}^n\), and begin with any basis of it.
The Gram–Schmidt Construction
Suppose \(\mathbf{v}_1,\dots,\mathbf{v}_k\) is a basis of \(E_{\lambda}\). The construction first keeps the direction of \(\mathbf{v}_1\), normalizing it to length one. At each later step, it subtracts from \(\mathbf{v}_j\) the parts pointing along the previously constructed unit vectors. The remainder is perpendicular to all of them; normalizing that remainder gives the next vector.
Proof. We prove the claims in the order needed to justify the construction. Since \(\mathbf{v}_1\) is a basis vector, it is nonzero. Hence \(\|\mathbf{u}_1\|=\|\mathbf{v}_1\|>0\), so \(\mathbf{q}_1\) is defined, has norm one, and belongs to \(E_{\lambda}\) because it is a scalar multiple of \(\mathbf{v}_1\).
Now suppose the construction has reached step \(j\), where \(2\le j\le k\), and that \(\mathbf{q}_1,\dots,\mathbf{q}_{j-1}\) are orthonormal vectors in \(E_{\lambda}\) spanning the same subspace as \(\mathbf{v}_1,\dots,\mathbf{v}_{j-1}\). Each term \((\mathbf{v}_j\cdot\mathbf{q}_i)\mathbf{q}_i\) belongs to \(E_{\lambda}\), as does \(\mathbf{v}_j\). Since an eigenspace is closed under linear combinations, their difference \(\mathbf{u}_j\) belongs to \(E_{\lambda}\). This is why orthogonalization preserves membership in the eigenspace: it uses only linear combinations of vectors already in that subspace.
For any \(r<j\), orthonormality of the earlier vectors gives $$ \mathbf{u}_j\cdot\mathbf{q}_r =\mathbf{v}_j\cdot\mathbf{q}_r -\sum_{i=1}^{j-1}(\mathbf{v}_j\cdot\mathbf{q}_i)(\mathbf{q}_i\cdot\mathbf{q}_r) =\mathbf{v}_j\cdot\mathbf{q}_r-(\mathbf{v}_j\cdot\mathbf{q}_r)=0. $$ So \(\mathbf{u}_j\) is perpendicular to each earlier \(\mathbf{q}_r\). It cannot be zero: if it were, the formula for \(\mathbf{u}_j\) would express \(\mathbf{v}_j\) as a linear combination of \(\mathbf{q}_1,\dots,\mathbf{q}_{j-1}\), hence of \(\mathbf{v}_1,\dots,\mathbf{v}_{j-1}\). That contradicts the linear independence of the original basis. Therefore \(\|\mathbf{u}_j\|>0\), and \(\mathbf{q}_j\) is defined. Dividing by its norm preserves membership in \(E_{\lambda}\), gives \(\|\mathbf{q}_j\|=1\), and preserves its perpendicularity to the earlier vectors.
Finally, the formula for \(\mathbf{u}_j\) shows that it is \(\mathbf{v}_j\) minus a combination of earlier \(\mathbf{q}_i\). Normalizing does not change the span. Thus, at each step, the first \(j\) constructed vectors span exactly the same subspace as the first \(j\) original basis vectors. At \(j=k\), the \(\mathbf{q}_i\) are orthonormal and span \(E_{\lambda}\). An orthonormal list is linearly independent: taking the dot product of \(\sum_i c_i\mathbf{q}_i=\mathbf{0}\) with \(\mathbf{q}_r\) gives \(c_r=0\) for every \(r\). Hence the list is an orthonormal basis of \(E_{\lambda}\). \(\square\)
Working Through the Subtractions
Worked Example: Orthonormalizing a Two-Dimensional Eigenspace
Let \(\mathbf{A}=\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}\). The vectors \(\mathbf{v}_1=(1,-1,0)^{\top}\) and \(\mathbf{v}_2=(1,0,-1)^{\top}\) are in \(E_0\), because each has coordinates summing to zero, and multiplication by \(\mathbf{A}\) gives zero. They are independent: if \(a\mathbf{v}_1+b\mathbf{v}_2=\mathbf{0}\), the second coordinate gives \(a=0\) and the third gives \(b=0\). Moreover, \(E_0\) consists exactly of vectors whose coordinates sum to zero, so these two independent vectors form a basis of that plane.
First, \(\|\mathbf{v}_1\|=\sqrt{1+1}=\sqrt{2}\), giving $$ \mathbf{q}_1=\frac{1}{\sqrt{2}}\begin{bmatrix}1\\-1\\0\end{bmatrix}. $$ The projection coefficient for the second vector is $$ \mathbf{v}_2\cdot\mathbf{q}_1=\frac{1}{\sqrt{2}}. $$ Subtracting that component gives $$ \mathbf{u}_2=\mathbf{v}_2-(\mathbf{v}_2\cdot\mathbf{q}_1)\mathbf{q}_1 =\begin{bmatrix}1\\0\\-1\end{bmatrix} -\frac12\begin{bmatrix}1\\-1\\0\end{bmatrix} =\begin{bmatrix}\tfrac12\\\tfrac12\\-1\end{bmatrix}. $$ Its squared norm is \(\tfrac14+\tfrac14+1=\tfrac32\), so its norm is \(\sqrt{6}/2\). Consequently, $$ \mathbf{q}_2=\frac{\mathbf{u}_2}{\|\mathbf{u}_2\|} =\frac{1}{\sqrt{6}}\begin{bmatrix}1\\1\\-2\end{bmatrix}. $$
Check: \(\mathbf{q}_1\cdot\mathbf{q}_2=(1-1+0)/\sqrt{12}=0\), and each vector has norm one. Both coordinate sums are zero, so multiplication by \(\mathbf{A}\) gives zero for each. They are an orthonormal basis of \(E_0\).
Worked Example: Three Vectors in a Repeated Eigenspace
Take \(\mathbf{A}=\begin{bmatrix}7&0&0&0\\0&7&0&0\\0&0&7&0\\0&0&0&2\end{bmatrix}\). Its eigenspace \(E_7\) consists of the vectors with fourth coordinate zero. Use the basis \(\mathbf{v}_1=(1,1,0,0)^{\top}\), \(\mathbf{v}_2=(1,0,1,0)^{\top}\), and \(\mathbf{v}_3=(0,1,1,0)^{\top}\). These vectors are independent: if their linear combination is zero, the first three coordinates give \(a+b=0\), \(a+c=0\), and \(b+c=0\); the first two imply \(b=c=-a\), and the third then gives \(-2a=0\), so \(a=b=c=0\).
The first vector has norm \(\sqrt{2}\), so \(\mathbf{q}_1=(1,1,0,0)^{\top}/\sqrt{2}\). For the second vector, \(\mathbf{v}_2\cdot\mathbf{q}_1=1/\sqrt{2}\), and hence $$ \mathbf{u}_2=\begin{bmatrix}1\\0\\1\\0\end{bmatrix} -\frac12\begin{bmatrix}1\\1\\0\\0\end{bmatrix} =\begin{bmatrix}\tfrac12\\-\tfrac12\\1\\0\end{bmatrix}, \qquad \|\mathbf{u}_2\|=\sqrt{\tfrac32}. $$ Thus \(\mathbf{q}_2=(1,-1,2,0)^{\top}/\sqrt{6}\).
For the third vector, the two projection coefficients are \(\mathbf{v}_3\cdot\mathbf{q}_1=1/\sqrt{2}\) and \(\mathbf{v}_3\cdot\mathbf{q}_2=1/\sqrt{6}\). Therefore $$ \mathbf{u}_3 =\begin{bmatrix}0\\1\\1\\0\end{bmatrix} -\frac12\begin{bmatrix}1\\1\\0\\0\end{bmatrix} -\frac16\begin{bmatrix}1\\-1\\2\\0\end{bmatrix} =\begin{bmatrix}-\tfrac23\\\tfrac23\\\tfrac23\\0\end{bmatrix}. $$ Its norm is \(2/\sqrt{3}\), giving \(\mathbf{q}_3=(-1,1,1,0)^{\top}/\sqrt{3}\).
Check: The three numerators have squared lengths \(2\), \(6\), and \(3\), respectively, so all three \(\mathbf{q}_i\) have norm one. Their pairwise dot products are \(1-1+0=0\), \(-1-1+2=0\), and \(-1+1+0=0\), with the corresponding normalizing factors. Each has fourth coordinate zero; therefore \(\mathbf{A}\mathbf{q}_i=7\mathbf{q}_i\) for every \(i\).
How Different Eigenspaces Fit Together
Orthogonalization within one eigenspace does not mix eigenvalues: every new vector remains in the same \(E_{\lambda}\). There is a related fact from the earlier result on orthogonality of distinct eigenspaces: for a real symmetric matrix, eigenspaces belonging to different eigenvalues are perpendicular. Thus, choosing an orthonormal basis inside each of several distinct eigenspaces produces a list that is orthonormal across the groups as well.
Proof. Vectors within any one chosen basis are orthonormal by construction. Vectors taken from two different chosen bases belong to eigenspaces for distinct eigenvalues, so their dot product is zero by the result on orthogonality of distinct eigenspaces. Every vector in the combined list is nonzero and belongs to an eigenspace, hence is an eigenvector. These facts establish the corollary. \(\square\)
Worked Example: Orthonormal Bases for Two Eigenspaces
Let \(\mathbf{A}=\begin{bmatrix}2&1\\1&2\end{bmatrix}\). Direct multiplication gives $$ \mathbf{A}\begin{bmatrix}1\\1\end{bmatrix} =\begin{bmatrix}3\\3\end{bmatrix} =3\begin{bmatrix}1\\1\end{bmatrix}, \qquad \mathbf{A}\begin{bmatrix}1\\-1\end{bmatrix} =\begin{bmatrix}1\\-1\end{bmatrix} =1\begin{bmatrix}1\\-1\end{bmatrix}. $$ The first vector spans \(E_3\): the equation \(\mathbf{A}(x,y)^{\top}=3(x,y)^{\top}\) reduces to \(y=x\). The second spans \(E_1\): the equation for eigenvalue \(1\) reduces to \(y=-x\). Each eigenspace is one-dimensional, so Gram–Schmidt here is normalization.
The two norms are both \(\sqrt{2}\), so set \(\mathbf{q}_3=(1,1)^{\top}/\sqrt{2}\) and \(\mathbf{q}_1=(1,-1)^{\top}/\sqrt{2}\). The check is \(\mathbf{q}_3\cdot\mathbf{q}_1=(1-1)/2=0\); both have norm one. Since normalization only rescales the eigenvectors, \(\mathbf{A}\mathbf{q}_3=3\mathbf{q}_3\) and \(\mathbf{A}\mathbf{q}_1=\mathbf{q}_1\). The combined list is orthonormal and consists of eigenvectors, one from each eigenspace.
This conclusion concerns the eigenspaces included in the construction. It does not, by itself, show that a chosen collection of eigenspaces spans all of \(\mathbb{R}^n\). Establishing a full orthonormal eigenbasis for every real symmetric matrix requires an additional existence argument; orthogonalization is the step that makes bases inside eigenspaces orthonormal, not the step that proves all vectors lie in those eigenspaces’ sum.
Common Pitfall: Subtract the Projection, Not a Multiple by Guesswork
In the first example, \(\mathbf{v}_1=(1,-1,0)^{\top}\) and \(\mathbf{v}_2=(1,0,-1)^{\top}\) have dot product \(1\). A tempting but wrong calculation is to subtract \((\mathbf{v}_2\cdot\mathbf{v}_1)\mathbf{v}_1=\mathbf{v}_1\) from \(\mathbf{v}_2\), obtaining \((0,1,-1)^{\top}\). This is not perpendicular to \(\mathbf{v}_1\), because $$ (0,1,-1)\cdot(1,-1,0)=-1\ne0. $$ The correct projection onto the line spanned by a nonunit vector \(\mathbf{v}_1\) is \(\frac{\mathbf{v}_2\cdot\mathbf{v}_1}{\mathbf{v}_1\cdot\mathbf{v}_1}\mathbf{v}_1\). Here the coefficient is \(1/2\), not \(1\). Subtracting \(\frac12\mathbf{v}_1\) gives \((1/2,1/2,-1)^{\top}\), whose dot product with \(\mathbf{v}_1\) is \(1/2-1/2=0\). In the Gram–Schmidt formula the earlier vectors \(\mathbf{q}_i\) are unit vectors, so the coefficient is simply \(\mathbf{v}_j\cdot\mathbf{q}_i\).
Check Your Understanding
Use the Gram–Schmidt formulas and verify each conclusion by checking norms, dot products, and eigenspace membership.
- For \(\mathbf{A}=\begin{bmatrix}4&0&0\\0&4&0\\0&0&1\end{bmatrix}\), orthonormalize the basis \((1,1,0)^{\top},(1,0,0)^{\top}\) of \(E_4\).
- Why can \(\mathbf{u}_j\) in the Gram–Schmidt construction not be zero when the starting list is a basis?
- Show directly that subtracting a linear combination of vectors in \(E_{\lambda}\) from another vector in \(E_{\lambda}\) stays in \(E_{\lambda}\).
- Normalize \((2,-1)^{\top}\) to obtain an orthonormal basis of its span.
- Explain why orthonormal bases chosen from two distinct eigenspaces of a real symmetric matrix are mutually perpendicular.
