From Nested Subspaces to Triangular Form
A matrix is upper triangular when every entry below its main diagonal is zero. This pattern has a geometric meaning: the first basis vector is sent into its own span, the first two basis vectors are sent into their span, and the same is true for each longer initial segment of the basis. These spans form a nested sequence of subspaces preserved by the operator. Conversely, if an operator preserves a nested sequence of subspaces of every dimension from \(1\) up to \(n\), a basis chosen to respect that sequence makes its matrix upper triangular.
Let \(V\) be a finite-dimensional vector space over \(\mathbb{F}\), where \(\mathbb{F}=\mathbb{R}\) or \(\mathbb{C}\), and let \(T:V\to V\) be linear. A subspace \(W\subseteq V\) is invariant under \(T\) if \(T(\mathbf{w})\in W\) for every \(\mathbf{w}\in W\), or equivalently, if \(T(W)\subseteq W\). A basis \(\mathcal{B}=(\mathbf{b}_1,\dots,\mathbf{b}_n)\) gives a matrix \([T]_{\mathcal{B}}\) whose \(j\)th column is \([T(\mathbf{b}_j)]_{\mathcal{B}}\). Thus the column convention makes the connection between invariance and matrix entries direct.
For an invariant flag, the inclusions are strict because the dimensions increase at every step. The space \(W_0=\{\mathbf{0}\}\) is also invariant, so it can be included at the start when useful. The flag records how the operator acts on increasingly large parts of the space, not merely how it acts on individual vectors.
The Equivalence and Its Proof
Proof. First suppose that \(\mathcal{B}=(\mathbf{b}_1,\dots,\mathbf{b}_n)\) is a basis in which \([T]_{\mathcal{B}}=(a_{ij})\) is upper triangular. For each \(k\), let \(W_k=\operatorname{span}\{\mathbf{b}_1,\dots,\mathbf{b}_k\}\). These subspaces are nested, have dimension \(k\), and \(W_n=V\). For each \(j\leq k\), upper triangularity gives \(a_{ij}=0\) for every \(i>j\), so \(T(\mathbf{b}_j)=\sum_{i=1}^{j}a_{ij}\mathbf{b}_i\in W_k\). Linearity then gives \(T(\mathbf{w})\in W_k\) for every \(\mathbf{w}\in W_k\). Hence every \(W_k\) is invariant, and the subspaces form an invariant flag.
Conversely, suppose \(W_1\subset\cdots\subset W_n=V\) is an invariant flag. Choose a nonzero vector \(\mathbf{b}_1\in W_1\). Inductively, for each \(k=2,\dots,n\), choose \(\mathbf{b}_k\in W_k\) that is not in \(W_{k-1}\). The vectors \(\mathbf{b}_1,\dots,\mathbf{b}_k\) form a basis of \(W_k\): they are independent because each newly chosen vector lies outside the span \(W_{k-1}\), and \(k\) independent vectors in the \(k\)-dimensional space \(W_k\) span it. In particular, \(\mathcal{B}=(\mathbf{b}_1,\dots,\mathbf{b}_n)\) is a basis of \(V\). Since \(W_j\) is invariant and \(\mathbf{b}_j\in W_j\), we have \(T(\mathbf{b}_j)\in W_j=\operatorname{span}\{\mathbf{b}_1,\dots,\mathbf{b}_j\}\). Therefore the \(j\)th column of \([T]_{\mathcal{B}}\) has zero entries in every row \(i>j\). This holds for each \(j\), so \([T]_{\mathcal{B}}\) is upper triangular. The two implications prove the equivalence. \(\square\)
The basis in the converse direction is called adapted to the flag: its first \(k\) vectors form a basis of \(W_k\). The construction explains why each subspace in the flag is needed. Preserving only the full space \(V\) gives no restriction, while preserving each successive \(W_k\) forces one more column of the matrix to have the required zero entries.
Reading the Flag from a Matrix
Worked Example: An Upper-Triangular Matrix Gives a Flag
Consider the operator \(T:\mathbb{R}^3\to\mathbb{R}^3\) with standard-basis matrix \(\mathbf{A}=\begin{bmatrix}2&-1&4\\0&3&5\\0&0&-1\end{bmatrix}\). Let \(\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3\) be the standard basis. Its columns give
Set \(W_1=\operatorname{span}\{\mathbf{e}_1\}\), \(W_2=\operatorname{span}\{\mathbf{e}_1,\mathbf{e}_2\}\), and \(W_3=\mathbb{R}^3\). For \(a\mathbf{e}_1\in W_1\), \(T(a\mathbf{e}_1)=2a\mathbf{e}_1\in W_1\). For \(a\mathbf{e}_1+b\mathbf{e}_2\in W_2\),
Finally, \(W_3\) is invariant because it is the whole space and \(T\) maps \(V\) into \(V\). Thus \(W_1\subset W_2\subset W_3\) is an invariant flag. The computations use each column of \(\mathbf{A}\), and the entries below the diagonal are zero exactly where the flag argument requires them to be.
Building a Triangular Matrix from a Flag
Worked Example: Changing to a Basis Adapted to the Flag
Define \(T:\mathbb{R}^3\to\mathbb{R}^3\) by its standard-basis matrix \(\mathbf{A}=\begin{bmatrix}2&1&0\\-2&1&2\\-4&-2&6\end{bmatrix}\). Consider the ordered list \(\mathcal{B}=(\mathbf{b}_1,\mathbf{b}_2,\mathbf{b}_3)\), where \(\mathbf{b}_1=(1,0,1)^{\top}\), \(\mathbf{b}_2=(0,1,1)^{\top}\), and \(\mathbf{b}_3=(0,0,1)^{\top}\).
The matrix whose columns are these vectors, and its inverse, are
Multiplying in both orders gives \(\mathbf{P}\mathbf{P}^{-1}=\mathbf{P}^{-1}\mathbf{P}=\mathbf{I}_3\), so the listed vectors are a basis. Applying \(\mathbf{A}\) to its basis vectors, or multiplying \(\mathbf{A}\mathbf{P}\), gives
The columns on the right are \(2\mathbf{b}_1\), \(\mathbf{b}_1+3\mathbf{b}_2\), and \(2\mathbf{b}_2+4\mathbf{b}_3\). Therefore, with \(W_1=\operatorname{span}\{\mathbf{b}_1\}\), \(W_2=\operatorname{span}\{\mathbf{b}_1,\mathbf{b}_2\}\), and \(W_3=\mathbb{R}^3\), the first line is mapped into \(W_1\), the first two basis vectors are mapped into \(W_2\), and \(W_3\) is invariant.
To obtain the matrix in the new basis, compute \(\mathbf{P}^{-1}\mathbf{A}\mathbf{P}\). The preceding product yields
This is upper triangular, as the theorem predicts. As a check on the coordinate calculation, its columns say \(T(\mathbf{b}_1)=2\mathbf{b}_1\), \(T(\mathbf{b}_2)=\mathbf{b}_1+3\mathbf{b}_2\), and \(T(\mathbf{b}_3)=2\mathbf{b}_2+4\mathbf{b}_3\), precisely the relations obtained by multiplying \(\mathbf{A}\) by the three vectors.
Invariant Lines and a Real Obstruction
Proof. If \(\mathbf{v}\) is an eigenvector, then \(T(\mathbf{v})=\lambda\mathbf{v}\) for some scalar \(\lambda\). For every \(c\in\mathbb{F}\), linearity gives \(T(c\mathbf{v})=c\lambda\mathbf{v}\in\operatorname{span}\{\mathbf{v}\}\), so the line is invariant. Conversely, if the line is invariant, then \(T(\mathbf{v})\) belongs to \(\operatorname{span}\{\mathbf{v}\}\), so \(T(\mathbf{v})=\lambda\mathbf{v}\) for some \(\lambda\in\mathbb{F}\). Since \(\mathbf{v}\ne\mathbf{0}\), this is the definition of an eigenvector. \(\square\)
Worked Example: A Real Rotation Has No Invariant Flag
Consider \(T:\mathbb{R}^2\to\mathbb{R}^2\) with matrix \(\mathbf{A}=\begin{bmatrix}0&-1\\1&0\end{bmatrix}\). An invariant flag in \(\mathbb{R}^2\) would have to begin with an invariant line. By the proposition, that would require a real eigenvector. For \(\mathbf{v}=(x,y)^{\top}\ne\mathbf{0}\), the eigenvector equation is
If \(y=0\), the second equation gives \(x=0\), contradicting \(\mathbf{v}\ne\mathbf{0}\). If \(y\ne0\), substitute \(x=\lambda y\) into the first equation to obtain \(-y=\lambda^2y\), hence \(\lambda^2=-1\). No real \(\lambda\) satisfies that equation. Thus there is no real invariant line and no real invariant flag. By the theorem, this operator has no upper-triangular matrix in any real basis.
This example shows why the scalar field belongs in statements about invariant flags. The same matrix entries can act on different vector spaces depending on whether the scalars are real or complex, and the existence of a suitable line can depend on that choice. The theorem itself makes no claim that every operator has a flag: it gives an exact condition for having an upper-triangular representation.
Common Pitfall: Using the Wrong Change-of-Basis Order
In the second worked example, \(\mathbf{P}\) has the new basis vectors as its columns. A common incorrect calculation is to use \(\mathbf{P}\mathbf{A}\mathbf{P}^{-1}\) as the matrix in that basis. For the displayed matrices, this gives
which is not upper triangular. The calculation does not contradict the theorem; it uses the wrong order. A coordinate column \(\mathbf{c}\) in the basis \(\mathcal{B}\) represents the standard-coordinate vector \(\mathbf{P}\mathbf{c}\). Applying \(T\) gives \(\mathbf{A}\mathbf{P}\mathbf{c}\), and converting back to \(\mathcal{B}\)-coordinates gives \(\mathbf{P}^{-1}\mathbf{A}\mathbf{P}\mathbf{c}\). Thus the matrix in the new basis is \(\mathbf{P}^{-1}\mathbf{A}\mathbf{P}\), which in this example is the verified upper-triangular matrix.
The practical test is to follow a coordinate vector through the two conversions: first from basis coordinates to standard coordinates using \(\mathbf{P}\), then through the operator using \(\mathbf{A}\), and finally back using \(\mathbf{P}^{-1}\). This order also makes the theorem useful in reverse: from a flag, choose a basis adapted to it, then check that each basis vector is sent into the span of itself and its predecessors.
Check Your Understanding
Use the definitions and the equivalence proved above to answer each question.
- For an upper-triangular matrix \(\mathbf{A}\in\mathbb{R}^{4\times4}\), describe the four subspaces in the flag obtained from its standard basis.
- Explain why invariance of \(W_j\) forces the entries below the diagonal in column \(j\) of an adapted-basis matrix to be zero.
- Suppose \(W_1\subset W_2\subset W_3\) is an invariant flag in a three-dimensional space. Describe how to choose \(\mathbf{b}_1,\mathbf{b}_2,\mathbf{b}_3\) so the resulting basis is adapted.
- Can an upper-triangular matrix have a zero diagonal entry and still arise from an invariant flag? Explain using the theorem’s condition.
- For the operator with matrix \(\begin{bmatrix}1&2\\0&3\end{bmatrix}\), identify an invariant line and verify its invariance by applying the operator to a spanning vector.
