Residual Orthogonality In Projection

The Residual and the Orthogonal Complement

When a vector \(\mathbf{b}\in\mathbb{R}^n\) is projected onto a subspace \(W\), the projection is the component that belongs to \(W\). The difference between the original vector and that component is called the residual. The key condition in this tutorial is that this residual is perpendicular to every vector in \(W\). It gives a practical way to check a proposed projection: the projected vector must lie in the subspace, and the residual must be orthogonal to the subspace.

We use the standard dot product \(\mathbf{x}\cdot\mathbf{y}=\mathbf{x}^{\top}\mathbf{y}\) on \(\mathbb{R}^n\). For a subspace \(W\subseteq\mathbb{R}^n\), its orthogonal complement is the set of vectors perpendicular to every vector in \(W\).

In finite-dimensional Euclidean space, the orthogonal decomposition theorem gives a unique expression \(\mathbf{b}=\mathbf{p}+\mathbf{r}\), where \(\mathbf{p}\in W\) and \(\mathbf{r}\in W^\perp\). The vector \(\mathbf{p}\) is the orthogonal projection of \(\mathbf{b}\) onto \(W\). We will prove the residual condition using the matrix characterization of orthogonal projection from the earlier material, and then prove that the two conditions also identify the projection uniquely.

Why the Residual Is Orthogonal

Suppose \(P\) is the matrix of the orthogonal projection onto \(W\). By the characterization of orthogonal projection matrices, \(P^{\top}=P\), \(P^2=P\), and the range of \(P\) is \(W\). The projected vector is \(P\mathbf{b}\), so its residual is \((I_n-P)\mathbf{b}\). The identities \(P^{\top}=P\) and \(P^2=P\) let us test this residual against any vector in the range of \(P\), which is exactly \(W\).

Proof. Let \(\mathbf{b}\in\mathbb{R}^n\), and let \(\mathbf{w}\in W\) be arbitrary. Since \(W\) is the range of \(P\), there is a vector \(\mathbf{z}\in\mathbb{R}^n\) such that \(\mathbf{w}=P\mathbf{z}\). Using symmetry and idempotence of \(P\), we obtain

because \((I_n-P)P=P-P^2=0\). Thus the residual has dot product zero with every \(\mathbf{w}\in W\). By the definition of \(W^\perp\), \(\mathbf{b}-P\mathbf{b}\in W^\perp\). \(\square\)

Membership and Orthogonality Characterize the Projection

The residual condition is not only necessary. Together with membership in \(W\), it is enough to identify the orthogonal projection. This characterization gives a direct verification method even when a projection matrix has not been computed. It is also important to keep both parts: a vector in \(W\) need not be the projection, and a residual orthogonal to \(W\) does not by itself say that the proposed projected vector belongs to \(W\).

Proof. First suppose \(\mathbf{q}=P\mathbf{b}\). The range of \(P\) is \(W\), so \(\mathbf{q}\in W\). The residual theorem just proved gives \(\mathbf{b}-\mathbf{q}=\mathbf{b}-P\mathbf{b}\in W^\perp\).

Conversely, suppose \(\mathbf{q}\in W\) and \(\mathbf{r}=\mathbf{b}-\mathbf{q}\in W^\perp\). Because \(P\) maps onto \(W\) and is idempotent, \(P\mathbf{q}=\mathbf{q}\). Also \(P\mathbf{r}\in W\), so \(\mathbf{r}\cdot P\mathbf{r}=0\) by \(\mathbf{r}\in W^\perp\). Symmetry and idempotence give

A vector has squared norm zero only if it is the zero vector, so \(P\mathbf{r}=\mathbf{0}\). Since \(\mathbf{b}=\mathbf{q}+\mathbf{r}\), linearity now gives \(P\mathbf{b}=P\mathbf{q}+P\mathbf{r}=\mathbf{q}\). Hence \(\mathbf{q}=P\mathbf{b}\), proving the reverse direction as well. \(\square\)

Testing a Projection with a Matrix

For a subspace given as the column space of a matrix, the residual condition can be checked against each column. If \(\mathbf{A}\) has columns that span \(W\), then a vector is perpendicular to all of \(W\) exactly when it is perpendicular to every column of \(\mathbf{A}\). In matrix notation, the condition is \(\mathbf{A}^{\top}(\mathbf{b}-\mathbf{q})=\mathbf{0}\). Indeed, the entries of this product are the dot products of the residual with the respective columns. This is a test for a candidate; it does not require the columns to be orthogonal.

Common Mistake: Checking Only Subspace Membership

A tempting but incorrect check is to verify only that a candidate \(\mathbf{q}\) lies in \(W\). In the matrix example, choose instead \(\mathbf{q}_{\mathrm{wrong}}=(1,1,2)\). It does lie in \(W\), because \(\mathbf{q}_{\mathrm{wrong}}=\mathbf{a}_1+\mathbf{a}_2\). But its residual for the same \(\mathbf{b}\) is

The first column already disproves orthogonality:

So \(\mathbf{q}_{\mathrm{wrong}}\) is not the projection. The correction is to check both conditions: the proposed vector must belong to \(W\), and its residual must be perpendicular to every generator of \(W\). Also take care with the subtraction order: the residual here is \(\mathbf{b}-\mathbf{q}\), not \(\mathbf{q}-\mathbf{b}\). Reversing the order changes the sign of every dot product but not whether those products equal zero.

Using the Condition Reliably

Residual orthogonality is useful because it turns a projection claim into checks involving subspace membership and dot products. For \(W=\operatorname{Col}\mathbf{A}\), checking \(\mathbf{A}^{\top}(\mathbf{b}-\mathbf{q})=\mathbf{0}\) checks all the generator directions at once. If the columns span \(W\), linearity ensures that zero dot product with each column gives zero dot product with every linear combination of those columns.

The condition also explains why the projection is determined uniquely. If two candidates both lie in \(W\) and both have residuals in \(W^\perp\), the projection characterization shows that each equals \(P\mathbf{b}\); therefore they are the same vector. This provides a proof of uniqueness without comparing their coordinates directly. The essential distinction is between the projected component, which belongs to \(W\), and the residual, which belongs to \(W^\perp\): together they decompose \(\mathbf{b}\) into orthogonal components.