From Column Generators to a Projection Matrix
An orthonormal list gives a direct formula for projecting onto its span. But a subspace can also be described by columns that are neither unit vectors nor mutually orthogonal. In that case, the coefficients of a vector in the column space are not obtained simply by taking its dot products with the columns. A matrix built from those dot products—the Gram matrix—corrects for the columns’ lengths and angles.
Throughout, scalars are real. Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\), with \(m,n\geq 1\), and write its columns as \(\mathbf{a}_1,\dots,\mathbf{a}_n\in\mathbb{R}^m\). The matrix has full column rank when its columns are linearly independent. Its column space is \(\operatorname{Col}\mathbf{A}=\operatorname{span}\{\mathbf{a}_1,\dots,\mathbf{a}_n\}\). We will find the matrix that projects each \(\mathbf{x}\in\mathbb{R}^m\) onto this subspace.
We use that theorem to identify the projection. Every vector \(\mathbf{y}\in\operatorname{Col}\mathbf{A}\) has the form \(\mathbf{A}\mathbf{c}\) for some coefficient vector \(\mathbf{c}\in\mathbb{R}^n\). For this to be the projection of \(\mathbf{x}\), the residual \(\mathbf{x}-\mathbf{A}\mathbf{c}\) must be orthogonal to every column of \(\mathbf{A}\). The entries of \(\mathbf{A}^{\top}(\mathbf{x}-\mathbf{A}\mathbf{c})\) are exactly the dot products of that residual with the columns. Thus the required condition is
The matrix \(\mathbf{A}^{\top}\mathbf{A}\) is square, but it is important to establish that it can be inverted. Full column rank is precisely what ensures this.
Proof. Suppose \((\mathbf{A}^{\top}\mathbf{A})\mathbf{c}=\mathbf{0}\) for some \(\mathbf{c}\in\mathbb{R}^n\). Multiplying on the left by \(\mathbf{c}^{\top}\) gives \(\mathbf{c}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{c}=0\). Since \(\mathbf{c}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{c}=(\mathbf{A}\mathbf{c})^{\top}(\mathbf{A}\mathbf{c})=\|\mathbf{A}\mathbf{c}\|^2\), this implies \(\mathbf{A}\mathbf{c}=\mathbf{0}\). The columns of \(\mathbf{A}\) are independent, so the only coefficients with \(\mathbf{A}\mathbf{c}=\mathbf{0}\) are \(\mathbf{c}=\mathbf{0}\). Therefore \(\mathbf{A}^{\top}\mathbf{A}\) has trivial null space, and the Invertible Matrix Theorem implies that this square matrix is invertible. \(\square\)
Deriving the Formula
Proof. Fix \(\mathbf{x}\in\mathbb{R}^m\). By the proposition, \(\mathbf{A}^{\top}\mathbf{A}\) is invertible. Define \(\mathbf{c}=(\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{x}\) and \(\mathbf{y}=\mathbf{A}\mathbf{c}\). By its definition, \(\mathbf{y}\in\operatorname{Col}\mathbf{A}\). The residual satisfies \[ \mathbf{A}^{\top}(\mathbf{x}-\mathbf{y}) =\mathbf{A}^{\top}\mathbf{x}-\mathbf{A}^{\top}\mathbf{A}\mathbf{c} =\mathbf{A}^{\top}\mathbf{x} -\mathbf{A}^{\top}\mathbf{A}(\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{x} =\mathbf{0}. \] Each entry of \(\mathbf{A}^{\top}(\mathbf{x}-\mathbf{y})\) is the dot product of the residual with one column of \(\mathbf{A}\). Hence the residual is orthogonal to each column, and therefore to every linear combination of the columns. Thus \(\mathbf{x}-\mathbf{y}\in(\operatorname{Col}\mathbf{A})^\perp\). By the definition and uniqueness of orthogonal projection, \(\mathbf{y}\) is the projection of \(\mathbf{x}\). Finally, \(\mathbf{y}=\mathbf{A}\mathbf{c}=\mathbf{A}(\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{x}=\mathbf{P}\mathbf{x}\), as required. \(\square\)
The full-column-rank hypothesis is doing two jobs: it makes the coefficients in \(\mathbf{A}\mathbf{c}\) unique, and it guarantees that the matrix being inverted is invertible. The matrix \(\mathbf{A}^{\top}\mathbf{A}\) records all pairwise dot products of the columns. When the columns are orthonormal, this Gram matrix is \(\mathbf{I}_n\), and the formula reduces to \(\mathbf{P}=\mathbf{A}\mathbf{A}^{\top}\). For general independent columns, the inverse Gram matrix accounts for their non-unit lengths and nonzero mutual dot products.
Computing a Projection Step by Step
To use the formula, it is often efficient to calculate \(\mathbf{A}^{\top}\mathbf{A}\), invert that smaller \(n\times n\) matrix, and then find the coefficient vector \(\mathbf{c}=(\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{x}\). The projected vector is \(\mathbf{A}\mathbf{c}\). This route can be clearer than forming every entry of the full \(m\times m\) projection matrix, especially when only one vector is being projected.
Worked Example: A Projection Matrix in \(\mathbb{R}^3\)
Find the projection matrix onto the column space of \(\mathbf{A}=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}\), and use it to project \(\mathbf{x}=\begin{bmatrix}1\\2\\0\end{bmatrix}\).
The columns are independent: if \(c_1(1,0,1)+c_2(0,1,1)=(0,0,0)\), the first coordinate gives \(c_1=0\), and the second gives \(c_2=0\). Thus the formula applies. First calculate the Gram matrix and its inverse:
For the inverse, multiplying \(\begin{bmatrix}2&1\\1&2\end{bmatrix}\) by \(\frac{1}{3}\begin{bmatrix}2&-1\\-1&2\end{bmatrix}\) gives \(\begin{bmatrix}1&0\\0&1\end{bmatrix}\). Now form the projection matrix, keeping the negative off-diagonal entries in the product:
Apply the formula to \(\mathbf{x}\). First, \(\mathbf{A}^{\top}\mathbf{x}=\begin{bmatrix}1\\2\end{bmatrix}\). Therefore the column coefficients are \[ \mathbf{c} =\frac{1}{3}\begin{bmatrix}2&-1\\-1&2\end{bmatrix} \begin{bmatrix}1\\2\end{bmatrix} =\frac{1}{3}\begin{bmatrix}0\\3\end{bmatrix} =\begin{bmatrix}0\\1\end{bmatrix}. \] The projection is \(\mathbf{A}\mathbf{c}=\begin{bmatrix}0\\1\\1\end{bmatrix}\). Directly, \(\mathbf{P}\mathbf{x}=\frac{1}{3}\begin{bmatrix}2-2\\-1+4\\1+2\end{bmatrix}=\begin{bmatrix}0\\1\\1\end{bmatrix}\), giving the same result.
Check: The residual is \(\mathbf{x}-\mathbf{A}\mathbf{c}=\begin{bmatrix}1\\1\\-1\end{bmatrix}\). Its dot products with the columns are \(1+0-1=0\) and \(0+1-1=0\). Thus it is orthogonal to the column space, as required.
Worked Example: Projecting onto a Different Two-Dimensional Subspace
Project \(\mathbf{x}=\begin{bmatrix}2\\0\\1\end{bmatrix}\) onto the column space of \(\mathbf{A}=\begin{bmatrix}1&0\\1&1\\0&1\end{bmatrix}\).
The columns \((1,1,0)\) and \((0,1,1)\) are independent: a linear combination equal to zero has first coordinate \(c_1=0\), then third coordinate \(c_2=0\). The Gram matrix and its inverse are
Next, \(\mathbf{A}^{\top}\mathbf{x}=\begin{bmatrix}2\\1\end{bmatrix}\), so \[ \mathbf{c} =\frac{1}{3}\begin{bmatrix}2&-1\\-1&2\end{bmatrix} \begin{bmatrix}2\\1\end{bmatrix} =\frac{1}{3}\begin{bmatrix}3\\0\end{bmatrix} =\begin{bmatrix}1\\0\end{bmatrix}. \] Consequently, the projected vector is \(\mathbf{A}\mathbf{c}=\begin{bmatrix}1\\1\\0\end{bmatrix}\).
Check: The residual is \(\begin{bmatrix}2\\0\\1\end{bmatrix}-\begin{bmatrix}1\\1\\0\end{bmatrix}=\begin{bmatrix}1\\-1\\1\end{bmatrix}\). Its dot products with the first and second columns are \(1-1+0=0\) and \(0-1+1=0\). Also, direct multiplication gives \(\mathbf{A}^{\top}\mathbf{A}\mathbf{c}=\begin{bmatrix}2\\1\end{bmatrix}=\mathbf{A}^{\top}\mathbf{x}\), confirming the coefficient equation.
When the Column Space Is the Whole Ambient Space
If \(\mathbf{A}\) is square and has full column rank, its columns form an independent list of \(m\) vectors in \(\mathbb{R}^m\), so they span \(\mathbb{R}^m\). In that case every vector already lies in the subspace onto which we project. The next calculation illustrates how the formula reflects this fact, while still finding the coefficients of the vector in the given columns.
Worked Example: A Square Full-Rank Matrix
Use the formula with \(\mathbf{A}=\begin{bmatrix}1&1\\0&2\end{bmatrix}\) to project \(\mathbf{x}=\begin{bmatrix}3\\-1\end{bmatrix}\) onto \(\operatorname{Col}\mathbf{A}\).
The columns are independent: if \(c_1(1,0)+c_2(1,2)=(0,0)\), the second coordinate gives \(2c_2=0\), and then the first gives \(c_1=0\). Compute
Multiplying to obtain the projection matrix gives \[ \mathbf{A}(\mathbf{A}^{\top}\mathbf{A})^{-1} =\frac{1}{4}\begin{bmatrix}4&0\\-2&2\end{bmatrix} =\begin{bmatrix}1&0\\-\tfrac12&\tfrac12\end{bmatrix}, \qquad \mathbf{P} =\begin{bmatrix}1&0\\-\tfrac12&\tfrac12\end{bmatrix} \begin{bmatrix}1&0\\1&2\end{bmatrix} =\begin{bmatrix}1&0\\0&1\end{bmatrix}. \] Thus \(\mathbf{P}\mathbf{x}=\begin{bmatrix}3\\-1\end{bmatrix}\). The coefficient calculation gives the same answer: \[ \mathbf{c} =\frac{1}{4}\begin{bmatrix}5&-1\\-1&1\end{bmatrix} \begin{bmatrix}3\\1\end{bmatrix} =\frac{1}{4}\begin{bmatrix}14\\-2\end{bmatrix} =\begin{bmatrix}\tfrac72\\-\tfrac12\end{bmatrix}, \qquad \mathbf{A}\mathbf{c} =\begin{bmatrix}\tfrac72-\tfrac12\\2(-\tfrac12)\end{bmatrix} =\begin{bmatrix}3\\-1\end{bmatrix}. \]
Check: The computed coefficient vector satisfies \(\mathbf{A}^{\top}\mathbf{A}\mathbf{c}=\begin{bmatrix}3\\1\end{bmatrix}=\mathbf{A}^{\top}\mathbf{x}\), and the residual is \(\mathbf{0}\). The projection is therefore the original vector.
Common Pitfall: Inverting the Wrong Matrix
A frequent error is to replace \((\mathbf{A}^{\top}\mathbf{A})^{-1}\) in the formula by \((\mathbf{A}\mathbf{A}^{\top})^{-1}\). For a tall \(m\times n\) matrix with \(m>n\), these matrices even have different sizes: \(\mathbf{A}^{\top}\mathbf{A}\) is \(n\times n\), while \(\mathbf{A}\mathbf{A}^{\top}\) is \(m\times m\). The second matrix may be singular, even when \(\mathbf{A}\) has full column rank.
For the matrix in the first worked example, the incorrect choice would require inverting \[ \mathbf{A}\mathbf{A}^{\top} =\begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix}. \] But \[ \begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix} \begin{bmatrix}1\\1\\-1\end{bmatrix} =\begin{bmatrix}0\\0\\0\end{bmatrix}, \] so this matrix is singular and has no inverse. The correct inverse is that of the \(2\times2\) Gram matrix \(\mathbf{A}^{\top}\mathbf{A}\), which is invertible by full column rank.
Also distinguish the coefficient vector from the projected vector. The vector \(\mathbf{c}=(\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{x}\) has \(n\) entries and tells us how to combine the columns. The projection itself is \(\mathbf{A}\mathbf{c}\), a vector in \(\mathbb{R}^m\). Stopping after finding \(\mathbf{c}\) does not yet give the projected vector unless the columns happen to be the standard basis.
The formula is useful whenever a subspace is given by an independent list of generators rather than an orthonormal one. Its central check is also its derivation: after computing \(\mathbf{y}\), verify that \(\mathbf{x}-\mathbf{y}\) has zero dot product with every column. That confirms that the residual is orthogonal to the entire column space.
Check Your Understanding
Use the full-column-rank formula where it applies, and check computed projections by testing residual orthogonality.
- For \(\mathbf{A}=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}\), calculate \(\mathbf{A}^{\top}\mathbf{A}\) and its inverse.
- Using that same matrix, find the projection of \(\begin{bmatrix}0\\1\\2\end{bmatrix}\) onto \(\operatorname{Col}\mathbf{A}\).
- Explain why full column rank implies that \(\mathbf{A}^{\top}\mathbf{A}\) is invertible.
- For \(\mathbf{A}=\begin{bmatrix}1&1\\1&0\\0&1\end{bmatrix}\), compute the coefficient vector for projecting \(\begin{bmatrix}1\\2\\0\end{bmatrix}\), then find the projected vector.
- Why does the expression \((\mathbf{A}\mathbf{A}^{\top})^{-1}\) fail for a full-column-rank \(3\times2\) matrix in general?
