Invariant and Reducing Subspaces
A subspace is invariant under an operator when applying the operator to any vector in the subspace keeps the result there. Invariance alone does not say what happens to vectors orthogonal to the subspace. A subspace is reducing when both it and its orthogonal complement are invariant. The distinction matters: an invariant subspace of a general operator need not be reducing, but for a finite-dimensional normal operator it always is.
Throughout, \(V\) is a finite-dimensional complex inner product space, and \(T:V\to V\) is linear. The inner product is linear in its first slot. The adjoint \(T^*\) is characterized by \(\langle T\mathbf{x},\mathbf{y}\rangle=\langle \mathbf{x},T^*\mathbf{y}\rangle\). An operator is normal if \(T^*T=TT^*\). For a subspace \(W\subseteq V\), its orthogonal complement is \(W^\perp=\{\mathbf{y}\in V:\langle\mathbf{y},\mathbf{w}\rangle=0\text{ for every }\mathbf{w}\in W\}\).
The equivalence in the definition follows from the adjoint relation. It is useful because block matrices often make invariance under \(T\) visible, while the adjoint relation explains what happens to the orthogonal complement.
Proof. First suppose \(W\) is invariant under \(T^*\). If \(\mathbf{y}\in W^\perp\) and \(\mathbf{w}\in W\), then \(\langle T\mathbf{y},\mathbf{w}\rangle=\langle\mathbf{y},T^*\mathbf{w}\rangle=0\), since \(T^*\mathbf{w}\in W\). Hence \(T\mathbf{y}\in W^\perp\), so \(W^\perp\) is invariant under \(T\). Conversely, suppose \(W^\perp\) is invariant under \(T\). For \(\mathbf{w}\in W\) and \(\mathbf{y}\in W^\perp\), \(\langle T^*\mathbf{w},\mathbf{y}\rangle=\langle\mathbf{w},T\mathbf{y}\rangle=0\), because \(T\mathbf{y}\in W^\perp\). Thus \(T^*\mathbf{w}\in (W^\perp)^\perp=W\); the last equality holds in a finite-dimensional inner product space. Therefore \(W^\perp\) is invariant under \(T\) exactly when \(W\) is invariant under \(T^*\). Combining this with invariance under \(T\) proves the equivalence. \(\square\)
Block Forms from Orthogonal Decompositions
In a finite-dimensional inner product space, an orthonormal basis of \(W\) can be extended to an orthonormal basis of \(V\). The additional basis vectors form an orthonormal basis of \(W^\perp\), giving the orthogonal decomposition \(V=W\oplus W^\perp\). Write the matrix of \(T\) using first the basis of \(W\), then the basis of \(W^\perp\). If \(W\) is invariant under \(T\), a vector in \(W\) has no \(W^\perp\) component after applying \(T\). Therefore the matrix has the form
Here \(\mathbf{A}\) describes the action from \(W\) to \(W\), \(\mathbf{C}\) describes the \(W^\perp\) component of the action on \(W^\perp\), and \(\mathbf{B}\) records the component that maps from \(W^\perp\) into \(W\). In an orthonormal basis, the matrix of \(T^*\) is the conjugate transpose:
Thus \(W^\perp\) is invariant under \(T\) exactly when \(\mathbf{B}=\mathbf{O}\). In that case the matrix is block diagonal. This gives a matrix description of reducing: relative to orthonormal bases of \(W\) and \(W^\perp\), a reducing subspace produces no off-diagonal blocks.
Worked Example: Invariant Does Not Always Mean Reducing
Consider \(T:\mathbb{C}^2\to\mathbb{C}^2\) with matrix \(\mathbf{A}=\begin{bmatrix}1&1\\0&2\end{bmatrix}\), and let \(W=\operatorname{span}\{\mathbf{e}_1\}\). Check invariance and determine whether \(W\) reduces \(T\).
The columns give \(T\mathbf{e}_1=\mathbf{e}_1\) and \(T\mathbf{e}_2=\mathbf{e}_1+2\mathbf{e}_2\). Hence \(T(W)\subseteq W\). Since the standard basis is orthonormal, \(W^\perp=\operatorname{span}\{\mathbf{e}_2\}\). But \(T\mathbf{e}_2=\mathbf{e}_1+2\mathbf{e}_2\notin W^\perp\), so \(W\) is not reducing.
The adjoint matrix and its action confirm this:
In the decomposition \(W\oplus W^\perp\), the original matrix already has the invariant-subspace block form \(\begin{bmatrix}1&1\\0&2\end{bmatrix}\); the nonzero upper-right entry records the failure of \(W^\perp\) to be invariant.
Check: \(T\mathbf{e}_1=(1,0)^\top\in W\), while \(T\mathbf{e}_2=(1,2)^\top\notin W^\perp\), as claimed. Also \(\mathbf{A}^*\mathbf{A}=\begin{bmatrix}1&1\\1&5\end{bmatrix}\) and \(\mathbf{A}\mathbf{A}^*=\begin{bmatrix}2&2\\2&4\end{bmatrix}\), so \(T\) is not normal.
Why Normality Forces Reduction
The key calculation uses a norm property of normal operators. If \(T\) is normal, then \(\|T\mathbf{x}\|=\|T^*\mathbf{x}\|\) for every \(\mathbf{x}\in V\): the squares of these norms are \(\langle T^*T\mathbf{x},\mathbf{x}\rangle\) and \(\langle TT^*\mathbf{x},\mathbf{x}\rangle\), respectively, and \(T^*T=TT^*\). To use this equality on an invariant subspace, we also use the finite-dimensional fact that a square matrix and its conjugate transpose have the same sum of squared entry magnitudes.
Proof. Choose orthonormal bases of \(W\) and \(W^\perp\), and write the matrix of \(T\) in block form. Invariance of \(W\) gives \(\mathbf{M}=\begin{bmatrix}\mathbf{A}&\mathbf{B}\\\mathbf{O}&\mathbf{C}\end{bmatrix}\), so \(\mathbf{M}^*=\begin{bmatrix}\mathbf{A}^*&\mathbf{O}\\\mathbf{B}^*&\mathbf{C}^*\end{bmatrix}\). Let \(\mathbf{u}_1,\ldots,\mathbf{u}_k\) be the coordinate standard basis for the \(W\)-block. For each \(j\), the coordinate vectors of \(T\mathbf{u}_j\) and \(T^*\mathbf{u}_j\) are, respectively, \((\mathbf{A}\mathbf{u}_j,\mathbf{0})\) and \((\mathbf{A}^*\mathbf{u}_j,\mathbf{B}^*\mathbf{u}_j)\). Normality gives equality of their squared norms, so \[ \|\mathbf{A}\mathbf{u}_j\|^2 = \|\mathbf{A}^*\mathbf{u}_j\|^2+\|\mathbf{B}^*\mathbf{u}_j\|^2. \] Sum over \(j=1,\ldots,k\). The sums \(\sum_j\|\mathbf{A}\mathbf{u}_j\|^2\) and \(\sum_j\|\mathbf{A}^*\mathbf{u}_j\|^2\) are equal: each is the sum of the squared magnitudes of all entries of \(\mathbf{A}\), with the first summing by columns and the second by rows. Therefore \(\sum_j\|\mathbf{B}^*\mathbf{u}_j\|^2=0\). Each term is nonnegative, so every \(\mathbf{B}^*\mathbf{u}_j=\mathbf{0}\). Since these vectors form a basis of the \(W\)-coordinate space, \(\mathbf{B}^*=\mathbf{O}\), and hence \(\mathbf{B}=\mathbf{O}\). The matrix of \(T\) is block diagonal, which means both \(W\) and \(W^\perp\) are invariant under \(T\). Thus \(W\) is reducing. \(\square\)
The finite-dimensional hypothesis enters in the choice of orthonormal bases and in the finite sum of squared entries in the proof. The conclusion is not that every subspace is invariant; it says that once a subspace is invariant for a normal operator, its orthogonal complement must also be invariant.
Worked Example: A Reducing Subspace of a Normal Operator
Let \(T:\mathbb{C}^3\to\mathbb{C}^3\) have matrix \(\mathbf{A}=\operatorname{diag}(2,i,-1)\), and let \(W=\operatorname{span}\{\mathbf{e}_1,\mathbf{e}_3\}\). Verify normality and find the block form for \(W\oplus W^\perp\).
The adjoint is \(\mathbf{A}^*=\operatorname{diag}(2,-i,-1)\). Since both matrices are diagonal, \(\mathbf{A}^*\mathbf{A}=\mathbf{A}\mathbf{A}^*=\operatorname{diag}(4,1,1)\), so \(T\) is normal. Also \(T\mathbf{e}_1=2\mathbf{e}_1\) and \(T\mathbf{e}_3=-\mathbf{e}_3\), so \(W\) is invariant. Its orthogonal complement is \(\operatorname{span}\{\mathbf{e}_2\}\), and \(T\mathbf{e}_2=i\mathbf{e}_2\) shows that this complement is invariant as well.
Order the orthonormal basis as \((\mathbf{e}_1,\mathbf{e}_3,\mathbf{e}_2)\), first listing a basis of \(W\). Then the matrix is \(\begin{bmatrix}2&0&0\\0&-1&0\\0&0&i\end{bmatrix}\). The \(W\)-block is \(\begin{bmatrix}2&0\\0&-1\end{bmatrix}\), the \(W^\perp\)-block is \([i]\), and both off-diagonal blocks are zero.
Check: Applying the block matrix to the first two coordinate vectors gives \(2\mathbf{e}_1\) and \(-\mathbf{e}_3\); applying it to the last gives \(i\mathbf{e}_2\). These are exactly the actions of \(T\) on the ordered basis, and the products \(\mathbf{A}^*\mathbf{A}\) and \(\mathbf{A}\mathbf{A}^*\) agree.
Block Forms in a Non-Coordinate Subspace
The decomposition need not use the standard coordinate axes. Any orthonormal basis adapted to \(W\oplus W^\perp\) gives the same structural conclusion: an invariant subspace of a normal operator has a block diagonal matrix in that basis. The entries of the blocks depend on the chosen bases, but whether the off-diagonal blocks vanish does not.
Worked Example: A Reducing Subspace Not Spanned by Coordinate Vectors
Define \(T:\mathbb{C}^3\to\mathbb{C}^3\) by \(\mathbf{A}=\operatorname{diag}(2,2,i)\). Consider \(\mathbf{q}_1=\frac{1}{\sqrt{2}}(1,1,0)^\top\), \(\mathbf{q}_2=\frac{1}{\sqrt{2}}(1,-1,0)^\top\), and \(\mathbf{q}_3=(0,0,1)^\top\). Let \(W=\operatorname{span}\{\mathbf{q}_1,\mathbf{q}_3\}\).
The vectors have unit norm, and their pairwise inner products are zero: \(\langle\mathbf{q}_1,\mathbf{q}_2\rangle=(1-1)/2=0\), while each inner product involving \(\mathbf{q}_3\) and either of the first two vectors is \(0\). Thus they form an orthonormal basis, with \(\mathbf{q}_2\) spanning \(W^\perp\). Direct calculation gives \(T\mathbf{q}_1=2\mathbf{q}_1\), \(T\mathbf{q}_3=i\mathbf{q}_3\), and \(T\mathbf{q}_2=2\mathbf{q}_2\). Therefore \(W\) and \(W^\perp\) are both invariant.
In the ordered basis \((\mathbf{q}_1,\mathbf{q}_3,\mathbf{q}_2)\), the matrix of \(T\) is \(\operatorname{diag}(2,i,2)\). Its first two basis vectors span \(W\), and its last spans \(W^\perp\), so this is the block form \(\begin{bmatrix}\begin{smallmatrix}2&0\\0&i\end{smallmatrix}&\mathbf{O}\\\mathbf{O}&[2]\end{bmatrix}\). The original matrix is normal because it is diagonal and its adjoint is \(\operatorname{diag}(2,2,-i)\).
Check: In standard coordinates, \(T\mathbf{q}_1=(2,2,0)^\top/\sqrt{2}=2\mathbf{q}_1\), \(T\mathbf{q}_2=(2,-2,0)^\top/\sqrt{2}=2\mathbf{q}_2\), and \(T\mathbf{q}_3=(0,0,i)^\top=i\mathbf{q}_3\). The stated block entries follow from these three images.
Common Pitfall: Assuming Invariance Is Enough
A common incorrect step is to see \(T(W)\subseteq W\) and conclude that \(T(W^\perp)\subseteq W^\perp\) without checking normality or the adjoint. The first worked example gives a concrete counterexample: \(W=\operatorname{span}\{\mathbf{e}_1\}\) is invariant, but \(T\mathbf{e}_2=\mathbf{e}_1+2\mathbf{e}_2\) has a nonzero component in \(W\). Thus the upper-right block is \(1\), not \(0\). The theorem applies only when \(T\) is normal; in that example, \(\mathbf{A}^*\mathbf{A}\ne\mathbf{A}\mathbf{A}^*\).
For an arbitrary operator, use the general block form \(\begin{bmatrix}\mathbf{A}&\mathbf{B}\\\mathbf{O}&\mathbf{C}\end{bmatrix}\) when \(W\) is invariant. Set \(\mathbf{B}=\mathbf{O}\) only when \(W^\perp\) is also invariant—for example, when the operator is normal and the hypotheses of the theorem hold.
Check Your Understanding
For each question, distinguish invariance of a subspace from invariance of its orthogonal complement.
- State the definition of a reducing subspace using \(T\), and give its equivalent definition using \(T\) and \(T^*\).
- If \(W\) is invariant under \(T\), what block of the matrix of \(T\), relative to \(W\oplus W^\perp\), must be zero?
- For the matrix \(\begin{bmatrix}1&1\\0&2\end{bmatrix}\), identify the invariant subspace and compute the image of a basis vector of its orthogonal complement.
- In the normal-operator proof, why does summing over an orthonormal basis of \(W\) force \(\mathbf{B}^*=\mathbf{O}\)?
- For \(\mathbf{A}=\operatorname{diag}(2,2,i)\) and \(W=\operatorname{span}\{(1,1,0)^\top,(0,0,1)^\top\}\), give an orthonormal basis of \(W^\perp\) and state the block matrix in an adapted orthonormal basis.
