Orthogonality and Fundamental Subspaces

Orthogonality in the Domain and Codomain

A matrix \(A\) with \(m\) rows and \(n\) columns represents a linear map from \(\mathbb{R}^n\) to \(\mathbb{R}^m\). Its row space and null space are subspaces of the domain, \(\mathbb{R}^n\); its column space and left null space are subspaces of the codomain, \(\mathbb{R}^m\). Orthogonality connects the spaces within each of these pairs. Keeping the ambient space in view prevents a common source of confusion: a row vector and a column vector of \(A\) usually live in different-dimensional spaces.

Throughout, use the Euclidean inner product \(x^T y\) on \(\mathbb{R}^k\). Two vectors \(x,y\in\mathbb{R}^k\) are orthogonal if \(x^T y=0\). A vector is orthogonal to a subspace if it is orthogonal to every vector in that subspace.

Definition: If \(S\) is a subspace of \(\mathbb{R}^k\), its orthogonal complement is \[ S^\perp=\{x\in\mathbb{R}^k:x^T s=0\text{ for every }s\in S\}. \] For \(A\in\mathbb{R}^{m\times n}\), the row space \(\operatorname{Row}(A)\) is the span of its rows, regarded as vectors in \(\mathbb{R}^n\); the column space \(\operatorname{Col}(A)\) is the span of its columns in \(\mathbb{R}^m\). The null space is \(\mathcal{N}(A)=\{x\in\mathbb{R}^n:Ax=0\}\), and the left null space is \(\mathcal{N}(A^T)=\{y\in\mathbb{R}^m:A^Ty=0\}\).

The row space and null space can be compared because both lie in \(\mathbb{R}^n\). Likewise, the column space and left null space both lie in \(\mathbb{R}^m\). These are the two orthogonal pairs among the four fundamental subspaces.

The Fundamental Orthogonality Relations

For any \(x\in\mathbb{R}^n\), the \(i\)th entry of \(Ax\) is the inner product of \(x\) with the \(i\)th row of \(A\). Thus \(Ax=0\) exactly when \(x\) is orthogonal to every row. The same reasoning applied to \(A^T y\) connects the left null space to the columns.

Theorem: For every real \(m\times n\) matrix \(A\), \[ \mathcal{N}(A)=\operatorname{Row}(A)^\perp\quad\text{in }\mathbb{R}^n, \qquad \mathcal{N}(A^T)=\operatorname{Col}(A)^\perp\quad\text{in }\mathbb{R}^m. \] Consequently, \(\operatorname{Row}(A)=\mathcal{N}(A)^\perp\) and \(\operatorname{Col}(A)=\mathcal{N}(A^T)^\perp\).

Proof. Write the rows of \(A\) as \(r_1^T,\ldots,r_m^T\), with each \(r_i\in\mathbb{R}^n\). For \(x\in\mathbb{R}^n\), \(Ax=0\) holds if and only if \(r_i^T x=0\) for every \(i\). That is equivalent to \(x\) being orthogonal to every row, and hence to every linear combination of the rows. Therefore \(\mathcal{N}(A)=\operatorname{Row}(A)^\perp\).

Write the columns of \(A\) as \(c_1,\ldots,c_n\in\mathbb{R}^m\). The entries of \(A^T y\) are \(c_j^T y\), for \(j=1,\ldots,n\). Thus \(A^T y=0\) if and only if \(y\) is orthogonal to every column, equivalently to their span. This proves \(\mathcal{N}(A^T)=\operatorname{Col}(A)^\perp\).

In a finite-dimensional Euclidean space, \((S^\perp)^\perp=S\) for every subspace \(S\). Applying this fact to the two equalities just proved gives the remaining identities. \(\square\)

Worked Example: A Rank-Deficient Matrix With Two Null Directions

Let \[ A=\begin{pmatrix}1&2&1\\2&4&2\end{pmatrix}. \] Find all four fundamental subspaces and verify the orthogonality relations.

The second row is twice the first, so the row space has dimension one and is \[ \operatorname{Row}(A)=\operatorname{span}\{(1,2,1)\}. \] To find the null space, solve \(Ax=0\). The two equations reduce to the single equation \(x_1+2x_2+x_3=0\). Setting \(x_2=s\) and \(x_3=t\) gives \(x=(-2s-t,s,t)\), so \[ \mathcal{N}(A)=\operatorname{span}\{(-2,1,0),(-1,0,1)\}. \] Both displayed basis vectors are orthogonal to the row vector: their inner products with \((1,2,1)\) are \(-2+2=0\) and \(-1+1=0\).

The columns are \((1,2)^T,(2,4)^T,(1,2)^T\). Therefore \[ \operatorname{Col}(A)=\operatorname{span}\{(1,2)^T\}. \] For \(y=(y_1,y_2)^T\), \(A^Ty=0\) reduces to \(y_1+2y_2=0\), so \[ \mathcal{N}(A^T)=\operatorname{span}\{(-2,1)^T\}. \] The vector \((-2,1)^T\) is orthogonal to the column generator since \((-2,1)(1,2)^T=-2+2=0\). The row and column spaces each have dimension one; the null and left null spaces have dimensions two and one, respectively.

Dimensions and Orthogonal Decompositions

The orthogonality identities also explain how the dimensions of the spaces fit together. Recall that the rank of \(A\) is the dimension of its row space, equivalently its column space, and that rank-nullity gives \(\dim\mathcal{N}(A)=n-\operatorname{rank}(A)\).

Theorem: If \(A\in\mathbb{R}^{m\times n}\) has rank \(r\), then \[ \dim\operatorname{Row}(A)=\dim\operatorname{Col}(A)=r,\qquad \dim\mathcal{N}(A)=n-r,\qquad \dim\mathcal{N}(A^T)=m-r. \] Moreover, \[ \mathbb{R}^n=\operatorname{Row}(A)\mathbin{\oplus^\perp}\mathcal{N}(A), \qquad \mathbb{R}^m=\operatorname{Col}(A)\mathbin{\oplus^\perp}\mathcal{N}(A^T). \] Here \(\oplus^\perp\) means that the two subspaces are orthogonal and together span the indicated space.

Proof. The row and column dimensions both equal \(r\) by the definition and equality of row rank and column rank. Rank-nullity for the map \(x\mapsto Ax\) gives \(\dim\mathcal{N}(A)=n-r\). Apply rank-nullity to \(A^T:\mathbb{R}^m\to\mathbb{R}^n\). Since \(A^T\) also has rank \(r\), \(\dim\mathcal{N}(A^T)=m-r\).

The first orthogonality identity shows \(\operatorname{Row}(A)\) and \(\mathcal{N}(A)\) are orthogonal. Their dimensions sum to \(r+(n-r)=n\), so their orthogonal sum has the full dimension of \(\mathbb{R}^n\) and equals \(\mathbb{R}^n\). The second identity gives orthogonality of \(\operatorname{Col}(A)\) and \(\mathcal{N}(A^T)\); their dimensions sum to \(r+(m-r)=m\), so their orthogonal sum equals \(\mathbb{R}^m\). \(\square\)

In particular, each vector in either ambient space has a unique expression as a sum of one vector from each subspace in its orthogonal pair. This describes the subspace structure without requiring an explicit choice of basis.

Worked Example: A Full-Column-Rank Matrix

Consider \[ A=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}. \] Its columns \(c_1=(1,0,1)^T\) and \(c_2=(0,1,1)^T\) are linearly independent, so \(\operatorname{rank}(A)=2\). The column space is their span. The two rows \((1,0),(0,1),(1,1)\) span \(\mathbb{R}^2\), so \(\operatorname{Row}(A)=\mathbb{R}^2\).

Solving \(Ax=0\) gives \(x_1=0\), \(x_2=0\), and \(x_1+x_2=0\). Hence \(\mathcal{N}(A)=\{0\}\), as the dimension formula predicts: \(n-r=2-2=0\).

To find the left null space, write \(y=(y_1,y_2,y_3)^T\). The equation \(A^Ty=0\) gives \(y_1+y_3=0\) and \(y_2+y_3=0\). Thus \(y=t(-1,-1,1)^T\), and \[ \mathcal{N}(A^T)=\operatorname{span}\{(-1,-1,1)^T\}. \] Its generator is orthogonal to both columns: \((-1,-1,1)c_1=-1+1=0\) and \((-1,-1,1)c_2=-1+1=0\). Its dimension is \(1=m-r=3-2\).

The Left Null Space and Consistency

The left null space gives a useful criterion for whether a proposed right-hand side \(b\in\mathbb{R}^m\) can be written as \(Ax\). By definition, the set of all such right-hand sides is the column space. Since the column space is the orthogonal complement of the left null space, consistency can be checked through inner products.

Result: For \(A\in\mathbb{R}^{m\times n}\) and \(b\in\mathbb{R}^m\), the equation \(Ax=b\) has a solution if and only if \[ y^T b=0\quad\text{for every }y\in\mathcal{N}(A^T). \]

Proof. The equation \(Ax=b\) has a solution exactly when \(b\in\operatorname{Col}(A)\), since \(\operatorname{Col}(A)\) consists of all vectors \(Ax\). By the fundamental orthogonality relation, \(\operatorname{Col}(A)=\mathcal{N}(A^T)^\perp\). Membership in this orthogonal complement means exactly that \(y^Tb=0\) for every \(y\in\mathcal{N}(A^T)\). \(\square\)

Worked Example: Detecting an Inconsistent System

Use \[ A=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}, \qquad b=\begin{pmatrix}1\\2\\4\end{pmatrix}. \] From the previous calculation, the left null space is spanned by \(y=(-1,-1,1)^T\). Its inner product with \(b\) is \[ y^Tb=(-1)(1)+(-1)(2)+(1)(4)=-1-2+4=1. \] This is not zero, so the consistency criterion shows that \(Ax=b\) has no solution.

The equations show the same conflict directly: the first two rows require \(x_1=1\) and \(x_2=2\), but then the third row requires \(x_1+x_2=3\), not \(4\). If instead \(b=(1,2,3)^T\), then \(y^Tb=-1-2+3=0\), and \(x=(1,2)^T\) verifies \(Ax=b\).

Common Mistake: Pairing the Wrong Spaces

It is incorrect to say that the null space is orthogonal to the column space. Those spaces usually lie in different ambient spaces: \(\mathcal{N}(A)\subseteq\mathbb{R}^n\), while \(\operatorname{Col}(A)\subseteq\mathbb{R}^m\). If \(m\ne n\), their vectors do not even have the same number of coordinates, so their Euclidean inner product is not defined.

The correction is to pair spaces in the same ambient space. In \(\mathbb{R}^n\), the row space is orthogonal to the null space. In \(\mathbb{R}^m\), the column space is orthogonal to the left null space. A second mistake is to treat orthogonality as equality: orthogonal spaces need not be the same space, and their dimensions are generally different. The dimension formulas specify exactly how they complement one another.

Key takeaway: For an \(m\times n\) matrix of rank \(r\), the row space and null space are orthogonal complements in \(\mathbb{R}^n\), with dimensions \(r\) and \(n-r\). The column space and left null space are orthogonal complements in \(\mathbb{R}^m\), with dimensions \(r\) and \(m-r\). The left null space also provides a complete test for consistency of \(Ax=b\).

Check Your Understanding

Use the ambient space of each subspace when answering.

For a matrix \(A\in\mathbb{R}^{4\times 6}\) of rank \(3\), what are the dimensions of its row space, column space, null space, and left null space?
State the orthogonal-complement identity that relates \(\mathcal{N}(A)\) to the rows of \(A\).
Why is the left null space, rather than the null space, used to test whether \(Ax=b\) is consistent?
If \(y\in\mathcal{N}(A^T)\), what condition must \(b\) satisfy for \(Ax=b\) to have a solution?
Explain why the row space and column space of a rectangular matrix are not generally an orthogonal pair.