Distances And Projection Errors

Distance as an Orthogonal Residual

When a vector does not lie in a subspace, its distance to that subspace measures how closely the subspace can approximate it. The distance is not the length of a particular coordinate difference chosen arbitrarily: it is the smallest possible error norm among all vectors in the subspace. The orthogonal projection identifies the vector that achieves this smallest error, and the residual from that projection gives the distance directly.

Throughout, work in \(\mathbb{R}^n\) with the Euclidean dot product \(\mathbf{u}\cdot\mathbf{v}=\mathbf{u}^{\top}\mathbf{v}\) and norm \(\|\mathbf{u}\|=\sqrt{\mathbf{u}\cdot\mathbf{u}}\). Let \(W\) be a subspace of \(\mathbb{R}^n\). For \(\mathbf{x}\in\mathbb{R}^n\), the distance from \(\mathbf{x}\) to \(W\) is the infimum of the errors \(\|\mathbf{x}-\mathbf{w}\|\) over all \(\mathbf{w}\in W\). Since \(W\) is finite-dimensional, the Best Approximation Theorem gives a unique vector \(\mathbf{p}\in W\) that minimizes this error. It is the orthogonal projection of \(\mathbf{x}\) onto \(W\), written \(\operatorname{proj}_W\mathbf{x}\); its residual is \(\mathbf{x}-\mathbf{p}\), which is orthogonal to every vector in \(W\).

The distinction between vector and number matters. A residual contains direction as well as size, while distance records only its length. Also, the Euclidean norm is built from squares: the squared distance is \(\|\mathbf{r}\|^2\), not a sum of the signed coordinates of \(\mathbf{r}\).

The Pythagorean Identity for Projection Errors

The key calculation compares the projection error with the error from any other choice in \(W\). It relies on the Pythagorean identity: when two vectors are orthogonal, the square of the norm of their sum is the sum of their squared norms.

Proof. By the definition of the Euclidean norm and the distributive law for the dot product, \(\|\mathbf{u}+\mathbf{v}\|^2=(\mathbf{u}+\mathbf{v})\cdot(\mathbf{u}+\mathbf{v})=\mathbf{u}\cdot\mathbf{u}+\mathbf{u}\cdot\mathbf{v}+\mathbf{v}\cdot\mathbf{u}+\mathbf{v}\cdot\mathbf{v}\). The dot product is symmetric over \(\mathbb{R}\), and the cross terms are zero because \(\mathbf{u}\cdot\mathbf{v}=0\). The remaining terms are \(\|\mathbf{u}\|^2+\|\mathbf{v}\|^2\). \(\square\)

Proof. Fix any \(\mathbf{w}\in W\). Since \(\mathbf{p}\) and \(\mathbf{w}\) both belong to the subspace, their difference \(\mathbf{p}-\mathbf{w}\) belongs to \(W\). The projection residual \(\mathbf{x}-\mathbf{p}\) is orthogonal to every vector in \(W\), so \((\mathbf{x}-\mathbf{p})\cdot(\mathbf{p}-\mathbf{w})=0\). Now split the error from \(\mathbf{w}\) as \(\mathbf{x}-\mathbf{w}=(\mathbf{x}-\mathbf{p})+(\mathbf{p}-\mathbf{w})\). The two summands are orthogonal, so the Pythagorean identity gives \(\|\mathbf{x}-\mathbf{w}\|^2=\|\mathbf{x}-\mathbf{p}\|^2+\|\mathbf{p}-\mathbf{w}\|^2\). The second term on the right is nonnegative, so every \(\mathbf{w}\in W\) has error at least \(\|\mathbf{x}-\mathbf{p}\|\). Taking \(\mathbf{w}=\mathbf{p}\) makes that second term zero, so the lower bound is attained and equals the infimum defining distance. If another \(\mathbf{w}\) attains the same distance, the identity forces \(\|\mathbf{p}-\mathbf{w}\|^2=0\), hence \(\mathbf{p}=\mathbf{w}\). This proves both the distance formula and uniqueness. \(\square\)

The identity gives more information than the statement that projection is best. The increase in squared error from choosing \(\mathbf{w}\) instead of \(\mathbf{p}\) is exactly \(\|\mathbf{p}-\mathbf{w}\|^2\). Thus, moving the chosen approximation farther from the projection adds a precise, nonnegative amount to its squared error.

Proof. The projection error identity says that \(\|\mathbf{x}-\mathbf{w}\|^2=d(\mathbf{x},W)^2+\|\mathbf{p}-\mathbf{w}\|^2\). Since the last term is nonnegative, \(\|\mathbf{x}-\mathbf{w}\|^2\ge d(\mathbf{x},W)^2\). Both norms are nonnegative, so taking square roots preserves the inequality. Equality holds exactly when \(\|\mathbf{p}-\mathbf{w}\|^2=0\), which is equivalent to \(\mathbf{w}=\mathbf{p}\). \(\square\)

Calculating and Comparing Distances

Common Pitfall: Adding Norms Instead of Squared Norms

The Pythagorean identity is an identity of squared norms. It does not say that the length of a sum of orthogonal vectors is the sum of their lengths. In the first example, take \(\mathbf{w}=\mathbf{0}\). The projection residual has norm \(\sqrt{2}\), and the displacement \(\mathbf{p}-\mathbf{0}\) has norm \(\sqrt{11}\). A common incorrect calculation is \(\|\mathbf{x}\|=\sqrt{2}+\sqrt{11}\). That would give the wrong error: the direct calculation gives \(\|\mathbf{x}\|=\sqrt{13}\), not \(\sqrt{2}+\sqrt{11}\). The correct calculation is \(\|\mathbf{x}\|^2=2+11=13\), and only then do we take the square root.

Another way to avoid confusion is to keep the vector equation and the scalar equation distinct. First split the error vector as \(\mathbf{x}-\mathbf{w}=(\mathbf{x}-\mathbf{p})+(\mathbf{p}-\mathbf{w})\). Then use orthogonality to add the squared norms. The two terms are perpendicular vectors, not two nonnegative numbers to add as lengths.

Using the Identity to Assess an Approximation

The projection error identity is useful whenever a subspace is used to approximate data or a vector. It tells us both the minimum achievable error and the exact penalty for a particular alternative. If an alternative \(\mathbf{w}\in W\) is close to the projection \(\mathbf{p}\), its squared error exceeds the minimum by only \(\|\mathbf{p}-\mathbf{w}\|^2\). If \(\mathbf{w}\) is far from \(\mathbf{p}\), that squared displacement is added in full. The identity applies whether \(W\) is a line, a plane, or any other subspace of \(\mathbb{R}^n\).

For practical calculations, the essential checks are that the proposed projection belongs to \(W\) and that its residual is orthogonal to \(W\). Once those conditions hold, the distance is the residual norm. To compare any other candidate, compute its displacement from the projection and add the square of that displacement to the squared distance. This avoids comparing complicated error vectors by inspection and makes equality conditions explicit.