The Best Approximation Theorem

Projection as a Best Approximation

Suppose a vector \(\mathbf{b}\) does not lie in a subspace \(W\). We may still want a vector in \(W\) that represents \(\mathbf{b}\) as closely as possible. The natural candidate is the orthogonal projection of \(\mathbf{b}\) onto \(W\): it is the part of \(\mathbf{b}\) that lies in \(W\), while the remaining part is perpendicular to \(W\). The best approximation theorem makes “as closely as possible” precise and proves that this candidate is the only closest vector in \(W\).

Throughout, work in \(\mathbb{R}^n\) with the standard dot product \(\mathbf{x}\cdot\mathbf{y}=\mathbf{x}^{\top}\mathbf{y}\) and norm \(\|\mathbf{x}\|=\sqrt{\mathbf{x}\cdot\mathbf{x}}\). For \(\mathbf{b}\in\mathbb{R}^n\) and \(\mathbf{w}\in W\), the distance from \(\mathbf{b}\) to \(\mathbf{w}\) is \(\|\mathbf{b}-\mathbf{w}\|\). A vector \(\mathbf{r}\) is perpendicular to \(W\) if \(\mathbf{r}\cdot\mathbf{w}=\mathbf{0}\) in the scalar sense—that is, \(\mathbf{r}\cdot\mathbf{w}=0\)—for every \(\mathbf{w}\in W\).

The orthogonal decomposition theorem gives a unique decomposition \(\mathbf{b}=\mathbf{p}+\mathbf{r}\) with \(\mathbf{p}\in W\) and \(\mathbf{r}\in W^\perp\). We use that result here: it ensures that the projection exists and is uniquely defined. The theorem below proves the further claim that this projection is not merely a vector in \(W\), but the unique vector in \(W\) closest to \(\mathbf{b}\).

The Pythagorean Identity

The key calculation compares the residual \(\mathbf{b}-\mathbf{p}\) with the difference between two vectors in the subspace. Because both \(\mathbf{p}\) and any \(\mathbf{w}\in W\) belong to \(W\), their difference \(\mathbf{p}-\mathbf{w}\) also belongs to \(W\). The residual is perpendicular to that difference.

Proof. By the definition of the norm and the distributive property of the dot product,

In \(\mathbb{R}^n\), \(\mathbf{x}\cdot\mathbf{y}=\mathbf{y}\cdot\mathbf{x}\). Since \(\mathbf{x}\cdot\mathbf{y}=0\), both cross terms are zero. The remaining terms are \(\|\mathbf{x}\|^2\) and \(\|\mathbf{y}\|^2\), which gives the stated identity. \(\square\)

The Best Approximation Theorem

Proof. By the definition of the projection, \(\mathbf{p}\in W\) and \(\mathbf{b}-\mathbf{p}\in W^\perp\). Fix any \(\mathbf{w}\in W\). Since \(W\) is a subspace, \(\mathbf{p}-\mathbf{w}\in W\), so \((\mathbf{b}-\mathbf{p})\cdot(\mathbf{p}-\mathbf{w})=0\). Decompose the difference from \(\mathbf{b}\) to \(\mathbf{w}\) as

The two vectors on the right are perpendicular. Applying the Pythagorean identity gives

A squared norm is nonnegative, so \(\|\mathbf{b}-\mathbf{w}\|^2\geq\|\mathbf{b}-\mathbf{p}\|^2\). Both norms are nonnegative, and squaring preserves their order; hence \(\|\mathbf{b}-\mathbf{w}\|\geq\|\mathbf{b}-\mathbf{p}\|\). Equality holds exactly when \(\|\mathbf{p}-\mathbf{w}\|^2=0\), which holds exactly when \(\mathbf{p}-\mathbf{w}=\mathbf{0}\), or \(\mathbf{w}=\mathbf{p}\). Thus \(\mathbf{p}\) is a minimizer and no other vector in \(W\) is a minimizer. \(\square\)

The identity is stronger than a comparison of distances: it tells exactly how much squared distance is added by choosing \(\mathbf{w}\) instead of the projection. The extra term is \(\|\mathbf{p}-\mathbf{w}\|^2\), which is zero only when the choice is the projection itself. This also explains why uniqueness follows from the geometry rather than from a particular formula for computing \(\mathbf{p}\).

Worked Examples

Why the Orthogonality Condition Matters

For any \(\mathbf{w}\in W\), the decomposition used in the proof has two parts: the residual \(\mathbf{b}-\mathbf{p}\), which points away from the subspace, and the in-subspace adjustment \(\mathbf{p}-\mathbf{w}\). Perpendicularity ensures their squared lengths add. Without that perpendicularity, the dot-product expansion has cross terms, and the distance need not be minimized at \(\mathbf{p}\). This is why merely choosing a vector in \(W\) is not enough; it must satisfy the projection condition.

The conclusion also applies when \(\mathbf{b}\in W\). In that case the decomposition is \(\mathbf{b}=\mathbf{b}+\mathbf{0}\), so \(\operatorname{proj}_W\mathbf{b}=\mathbf{b}\). The minimum distance is \(0\), and the theorem says no other vector in \(W\) has distance \(0\) from \(\mathbf{b}\). At the other boundary case \(W=\{\mathbf{0}\}\), the only available vector is \(\mathbf{0}\), which is necessarily the unique minimizer; the theorem’s comparison remains valid.

A Common Pitfall

A frequent error is to subtract the squared correction term instead of adding it. In the line example, \(\mathbf{b}=(3,1)\), \(\mathbf{p}=(2,2)\), and \(\mathbf{w}=(3,3)\). The incorrect calculation would claim

But direct substitution gives \(\mathbf{b}-\mathbf{w}=(0,-2)\), so \(\|\mathbf{b}-\mathbf{w}\|^2=4\), not \(0\). The correct Pythagorean identity uses a plus sign: \(2+2=4\). Geometrically, moving from the projection to another vector in \(W\) adds a perpendicular displacement to the residual, so it increases the squared distance rather than subtracting from it.