Projection Onto An Orthogonal List

From Projection onto a Line to an Orthogonal List

For a single nonzero vector \(\mathbf{u}\in\mathbb{R}^n\), the projection of \(\mathbf{b}\) onto the line spanned by \(\mathbf{u}\) is the vector on that line whose difference from \(\mathbf{b}\) is perpendicular to the line. If a subspace is spanned by several mutually orthogonal vectors, the same idea applies: find a vector in the span so that the leftover part is perpendicular to every generator. Orthogonality makes the calculation especially direct, because each generator contributes its own coefficient without interfering with the others.

Throughout, vectors are columns in \(\mathbb{R}^n\), with dot product \(\mathbf{u}\cdot\mathbf{v}=\mathbf{u}^{\top}\mathbf{v}\). A finite list \(\mathbf{u}_1,\dots,\mathbf{u}_p\) is orthogonal if \(\mathbf{u}_i\cdot\mathbf{u}_j=0\) whenever \(i\ne j\). It is an orthogonal list of nonzero vectors if it is orthogonal and every \(\mathbf{u}_i\ne\mathbf{0}\). Its span is the set of all linear combinations of its vectors.

In other words, the projected vector belongs to the subspace, and the residual is perpendicular to the subspace. For a span, it is enough to check perpendicularity against each generator: if a vector is perpendicular to every \(\mathbf{u}_i\), then by linearity of the dot product it is perpendicular to every linear combination of them.

The Projection Formula

Proof. Define \(c_i=(\mathbf{b}\cdot\mathbf{u}_i)/(\mathbf{u}_i\cdot\mathbf{u}_i)\) and \(\mathbf{w}=\sum_{i=1}^{p}c_i\mathbf{u}_i\). Since \(\mathbf{w}\) is a linear combination of the generators, \(\mathbf{w}\in W\). For each \(k\), orthogonality gives \(\mathbf{u}_k\cdot\mathbf{u}_i=0\) when \(i\ne k\), so \[ \mathbf{u}_k\cdot(\mathbf{b}-\mathbf{w}) =\mathbf{u}_k\cdot\mathbf{b}-\sum_{i=1}^{p}c_i(\mathbf{u}_k\cdot\mathbf{u}_i) =\mathbf{u}_k\cdot\mathbf{b}-c_k(\mathbf{u}_k\cdot\mathbf{u}_k)=0. \] Thus the residual is perpendicular to every generator and hence to every vector in \(W\). This proves that \(\mathbf{w}\) satisfies the definition of the projection. To verify uniqueness, suppose \(\mathbf{w}_1,\mathbf{w}_2\in W\) both satisfy the definition. Then \(\mathbf{w}_1-\mathbf{w}_2\in W\), and \(\mathbf{w}_1-\mathbf{w}_2=(\mathbf{b}-\mathbf{w}_2)-(\mathbf{b}-\mathbf{w}_1)\in W^\perp\). A vector \(\mathbf{z}\) in both \(W\) and \(W^\perp\) satisfies \(\mathbf{z}\cdot\mathbf{z}=0\), which forces \(\mathbf{z}=\mathbf{0}\). Therefore \(\mathbf{w}_1=\mathbf{w}_2\), proving the formula gives the unique projection. \(\square\)

The denominator \(\mathbf{u}_i\cdot\mathbf{u}_i=\|\mathbf{u}_i\|^2\) is the squared length of the generator. It is positive because \(\mathbf{u}_i\ne\mathbf{0}\). If the list is orthonormal, meaning its vectors are mutually orthogonal and each has length \(1\), the denominator is \(1\), so each coefficient reduces to \(\mathbf{b}\cdot\mathbf{u}_i\). The formula above does not require the vectors to have unit length.

Proof. Suppose \(c_1\mathbf{u}_1+\cdots+c_p\mathbf{u}_p=\mathbf{0}\). Dot both sides with \(\mathbf{u}_k\). All terms except the \(k\)-th vanish by orthogonality, giving \(c_k(\mathbf{u}_k\cdot\mathbf{u}_k)=0\). Since \(\mathbf{u}_k\ne\mathbf{0}\), its squared length is positive, so \(c_k=0\). This holds for each \(k\), and therefore every coefficient is zero. \(\square\)

This result also explains why the coefficients in the formula specify one representation of the projected vector in the given list: a linear combination of an independent list has unique coefficients. If the list is empty, its span is \(\{\mathbf{0}\}\); the corresponding projection is \(\mathbf{0}\) and the residual is \(\mathbf{b}\). The formula is then interpreted as an empty sum.

Calculating a Projection and Residual

What the Formula Depends On

The formula works because each generator is perpendicular to every other generator. When the residual is dotted with \(\mathbf{u}_k\), all terms involving \(\mathbf{u}_i\) for \(i\ne k\) disappear. This isolates exactly one coefficient. If the generators are not orthogonal, those cross terms need not vanish, so using the same separate coefficient formula is not justified.

Nonzero vectors are also essential. If a generator were zero, its squared length would be zero, making the stated coefficient undefined. A zero vector contributes nothing to a span, so it should be omitted before applying the formula. For an orthogonal list of nonzero vectors, each denominator is positive, and the computation is valid.

Common Mistake: Forgetting the Squared Length

A frequent error is to use \(\mathbf{b}\cdot\mathbf{u}_i\) as the coefficient even when \(\mathbf{u}_i\) is not a unit vector. For instance, take the orthogonal vectors \(\mathbf{u}_1=(2,0)\), \(\mathbf{u}_2=(0,3)\), and \(\mathbf{b}=(3,4)\). The incorrect coefficients would be \(\mathbf{b}\cdot\mathbf{u}_1=6\) and \(\mathbf{b}\cdot\mathbf{u}_2=12\), giving the incorrect vector \(6(2,0)+12(0,3)=(12,36)\). It is not even a plausible projection of \((3,4)\): its residual is \((-9,-32)\), which is not perpendicular to either generator.

The correction is to divide each dot product by the corresponding squared length: \[ \frac{\mathbf{b}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1} =\frac{6}{4}=\frac32,\qquad \frac{\mathbf{b}\cdot\mathbf{u}_2}{\mathbf{u}_2\cdot\mathbf{u}_2} =\frac{12}{9}=\frac43. \] Then the projected vector is \[ \frac32(2,0)+\frac43(0,3)=(3,0)+(0,4)=(3,4)=\mathbf{b}. \] The residual is \((0,0)\), whose dot product with each generator is zero. The denominator accounts for the length of the generator; it disappears only when that length is \(1\).

To check any calculation, verify two facts separately: the proposed projection is a linear combination of the generators, and the residual has dot product zero with each generator. These checks directly test the defining conditions, and they can expose an arithmetic error even when the expression appears to have the right form.