Orthogonal Projections Onto Closed Subspaces

Closest Points and Orthogonality

Approximating a function by a simpler one is often a minimization problem: among all candidates in a specified subspace, find the one with the smallest \(L^2\) error. The key fact is that in a Hilbert space this minimizer exists whenever the subspace is closed, and it is characterized by an orthogonality condition. This converts a global minimization problem into equations involving inner products.

Let \((X,\mathcal{M},\mu)\) be any measure space. We use \(L^2(\mu)\) for the Hilbert space of equivalence classes, modulo equality \(\mu\)-a.e., of complex-valued measurable functions \(f\) such that \(\int_X |f|^2\,d\mu<\infty\). Its inner product convention is \(\langle f,g\rangle=\int_X f\overline{g}\,d\mu\), linear in the first argument, and its norm is \(\|f\|_2=\langle f,f\rangle^{1/2}\). The theorem below applies more generally to any real or complex Hilbert space \(H\); the completeness of \(L^2(\mu)\) is the standard Hilbert-space result established earlier.

The projection theorem says that a closed subspace has exactly one best approximation to every vector. Its proof uses completeness and the parallelogram identity; it does not require the subspace to be finite-dimensional.

The Projection Theorem

Proof. Fix \(f\in H\) and let \(d=\inf_{q\in M}\|f-q\|\). This is a finite nonnegative real number because \(0\in M\), so \(0\le d\le\|f\|\). By the definition of infimum, choose \(p_n\in M\) with \(d\le\|f-p_n\|<d+1/n\). The parallelogram identity, applied to \(f-p_n\) and \(f-p_k\), gives \(\|p_n-p_k\|^2=2\|f-p_n\|^2+2\|f-p_k\|^2-4\|f-(p_n+p_k)/2\|^2\). Because \(M\) is a linear subspace, \((p_n+p_k)/2\in M\), so the last norm is at least \(d\). Consequently, \(\|p_n-p_k\|^2\le 2\|f-p_n\|^2+2\|f-p_k\|^2-4d^2\), which tends to zero as \(n,k\to\infty\). Thus \((p_n)\) is Cauchy. Completeness of \(H\) gives a limit \(p\in H\), and closedness of \(M\) gives \(p\in M\). Norm continuity yields \(\|f-p\|=d\), proving existence of a minimizer.

Set \(r=f-p\). For any \(m\in M\) and scalar \(z\), \(p+zm\in M\). Minimality of \(p\) therefore gives \(\|r-zm\|^2\ge\|r\|^2\). If \(H\) is real, expanding this inequality for both positive and negative real \(z\) forces \(\langle r,m\rangle=0\). If \(H\) is complex, the expansion is \(\|r-zm\|^2-\|r\|^2=|z|^2\|m\|^2-2\operatorname{Re}(z\langle m,r\rangle)\). Taking real \(z=t\) of either sign shows \(\operatorname{Re}\langle m,r\rangle=0\); taking \(z=it\) of either sign shows \(\operatorname{Im}\langle m,r\rangle=0\). Thus \(r\perp M\) in either case.

Conversely, suppose \(p\in M\) and \(f-p\perp M\). For any \(q\in M\), \(p-q\in M\), so \(f-p\perp p-q\). Expanding the squared norm, with the cross terms zero, gives \(\|f-q\|^2=\|(f-p)+(p-q)\|^2=\|f-p\|^2+\|p-q\|^2\ge\|f-p\|^2\). Hence \(p\) is a best approximation. If \(p_1,p_2\in M\) are both best approximations, the characterization gives \(f-p_1\perp M\) and \(f-p_2\perp M\). Their difference \(p_1-p_2\) belongs to \(M\) and is orthogonal to \(M\), so \(\|p_1-p_2\|^2=0\), and \(p_1=p_2\). This proves existence, the characterization, and uniqueness. \(\square\)

Proof. The theorem gives \(f-P_M f\perp M\). For \(f,g\in H\), the vector \(P_M f+P_M g\) lies in \(M\), and \((f+g)-(P_M f+P_M g)\) is orthogonal to \(M\). By the theorem's uniqueness characterization, \(P_M(f+g)=P_Mf+P_Mg\). For a scalar \(c\), \(cP_Mf\in M\) and \(cf-cP_Mf\perp M\), so \(P_M(cf)=cP_Mf\); this also holds for \(c=0\). If \(p\in M\), its residual \(p-p=0\) is orthogonal to \(M\), giving \(P_Mp=p\). Finally, \(P_Mf\) and \(f-P_Mf\) are orthogonal, so \(\|f\|^2=\|P_Mf\|^2+\|f-P_Mf\|^2\), which implies \(\|P_Mf\|\le\|f\|\). \(\square\)

Computing Projections in Finite Dimensions

Suppose \(M\) is spanned by a finite linearly independent list \(v_1,\dots,v_N\) in a Hilbert space. To find the best approximation \(p=\sum_{j=1}^N c_jv_j\) to \(f\), require \(f-p\perp v_k\) for each \(k\). With the convention that the inner product is linear in the first argument, these conditions are the normal equations \(\sum_{j=1}^N c_j\langle v_j,v_k\rangle=\langle f,v_k\rangle\), for \(k=1,\dots,N\). The matrix of this system is invertible: if coefficients \(c_j\) solve the homogeneous system, then \(w=\sum_jc_jv_j\) is orthogonal to every \(v_k\), hence to \(w\), so \(\|w\|^2=0\). Linear independence then gives \(c_j=0\) for every \(j\). Thus the system has a unique solution, and its residual is orthogonal to \(M\), so the projection theorem's characterization proves it is the best approximation.

When the spanning vectors are already orthogonal and nonzero, the equations separate. The coefficient of \(v_j\) is \(\langle f,v_j\rangle/\|v_j\|^2\). Orthogonality is essential to this simple coefficient-by-coefficient formula; for a nonorthogonal list, the coupled normal equations must be solved.

Why Normalization Matters

A common mistake is to project onto the constant functions by integrating \(f\) and using that integral directly as the constant. The coefficient must account for the inner product of the constant basis vector with itself. On \([0,2]\) with Lebesgue measure, the constant projection of \(f(x)=x\) is \(c=\langle x,1\rangle/\|1\|_2^2\), not just \(\langle x,1\rangle\). Here \(\langle x,1\rangle=\int_0^2x\,dx=2\), while \(\|1\|_2^2=\int_0^2 1\,dx=2\); hence \(c=2/2=1\). The incorrect choice \(c=2\) has residual integral \(\int_0^2(x-2)\,dx=2-4=-2\ne0\), so it is not orthogonal to constants. The correct choice has residual integral \(\int_0^2(x-1)\,dx=2-2=0\).

For a finite-dimensional approximation, the orthogonality equations provide both a practical computation and a decisive check: after finding the coefficients, verify the residual against each spanning vector. Once that condition holds, the Pythagorean identity shows that every other candidate has squared error equal to the projection's squared error plus a nonnegative squared distance within the subspace. No candidate can do better.