Projection Approximation Error In Nested Spaces

Approximation Error as a Population Distance

A linear prediction space can be too small to represent the regression pattern in a population. Its approximation error measures the cost of that restriction before accounting for how accurately the space can be estimated from a sample. Under squared loss, this cost has a precise geometric form: it is the squared distance from the target regression function to the space.

Let \(X\) be a covariate, \(Y\) a response, and assume \(E[Y^2]<\infty\). Take \(X\) to have values in a measurable space for which the conditional mean has a measurable version \(m(x)=E[Y\mid X=x]\). A prediction function \(g\) assigns the value \(g(X)\) to a new covariate value. Its population squared risk is \(R(g)=E[(Y-g(X))^2]\). We measure distances between functions using the \(L_2(P_X)\) norm, \(\|h\|_{P_X}^2=E[h(X)^2]\); functions equal with probability one are identified.

The reference risk \(R(m)\) is the risk of predicting with the conditional mean. It need not be zero: random variation in \(Y\) given \(X\) can remain even when the regression function is known exactly. The first result separates that variation from the cost of using a different prediction function.

Proof. Write \(Y-g(X)=(Y-m(X))+(m(X)-g(X))\) and expand the square. The cross term has expectation zero because \(m(X)-g(X)\) is a function of \(X\) and \(E[Y-m(X)\mid X]=0\). More explicitly, conditional expectation gives $$ E[(Y-m(X))(m(X)-g(X))] =E[(m(X)-g(X))E[Y-m(X)\mid X]]=0. $$ Therefore \(R(g)=E[(Y-m(X))^2]+E[(m(X)-g(X))^2]\). The first term is \(E[\operatorname{Var}(Y\mid X)]\), which equals \(R(m)\), proving the identity. Subtracting \(R(m)\) and taking the infimum over \(\mathcal S\) gives the approximation-error formula. \(\square\)

Projection in a Finite-Dimensional Linear Space

Suppose \(\mathcal S\) is spanned by \(d\) linearly independent functions \(\phi_1,\ldots,\phi_d\) in \(L_2(P_X)\). A candidate in this space has the form \(g_\beta(x)=\sum_{j=1}^d\beta_j\phi_j(x)\). Define the Gram matrix \(G\) and vector \(c\) by \(G_{ij}=E[\phi_i(X)\phi_j(X)]\) and \(c_i=E[\phi_i(X)m(X)]\). Linear independence in \(L_2(P_X)\) makes \(G\) positive definite, so it is invertible.

Proof. The residual \(r=m-g_{\beta^*}\) is orthogonal to every basis function: for each \(i\), $$ E[\phi_i(X)r(X)]=c_i-\sum_{j=1}^dG_{ij}\beta_j^*=c_i-(G\beta^*)_i=0. $$ For any other coefficient vector \(\beta\), \(g_\beta-g_{\beta^*}\) belongs to \(\mathcal S\), and hence is orthogonal to \(r\). Expanding the squared norm gives $$ \|m-g_\beta\|_{P_X}^2 =\|r-(g_\beta-g_{\beta^*})\|_{P_X}^2 =\|r\|_{P_X}^2+\|g_\beta-g_{\beta^*}\|_{P_X}^2. $$ Thus \(g_{\beta^*}\) minimizes the distance, uniquely up to equality in \(L_2(P_X)\). Also, \(E[m g_{\beta^*}]=(\beta^*)^\mathsf{T}c=c^\mathsf{T}G^{-1}c\), and \(E[g_{\beta^*}^2]=(\beta^*)^\mathsf{T}G\beta^*=c^\mathsf{T}G^{-1}c\). Expanding \(E[(m-g_{\beta^*})^2]\) therefore yields \(E[m^2]-c^\mathsf{T}G^{-1}c\), as claimed. \(\square\)

Nested Spaces and Decreasing Approximation Error

A sieve is often organized as a sequence of nested spaces, \(\mathcal S_1\subseteq\mathcal S_2\subseteq\cdots\). The best approximation in a larger space can always use a function from the smaller space, so enlarging the available set cannot worsen the best population approximation.

Proof. Each approximation error is an infimum of squared norms, so it is nonnegative. Since \(\mathcal S_k\) is contained in \(\mathcal S_{k+1}\), minimizing \(\|m-g\|_{P_X}^2\) over \(\mathcal S_{k+1}\) cannot give a larger value than minimizing over \(\mathcal S_k\). For the identity, the projection residual \(m-g_{k+1}^*\) is orthogonal to every element of \(\mathcal S_{k+1}\), including \(g_{k+1}^*-g_k^*\). Consequently, $$ \|m-g_k^*\|_{P_X}^2 =\|(m-g_{k+1}^*)+(g_{k+1}^*-g_k^*)\|_{P_X}^2 =\|m-g_{k+1}^*\|_{P_X}^2+\|g_{k+1}^*-g_k^*\|_{P_X}^2. $$ The two squared distances are the stated approximation errors. \(\square\)

Interpretation and a Common Pitfall

Approximation error describes a population limitation: even with the best possible choice of coefficients in a given space, the space may not contain a close enough representation of \(m\). It is distinct from error caused by estimating coefficients using finite data. The nested-space result concerns only these best population approximations; it does not say that fitting a larger model to a particular sample will improve its performance on new data.

• Common mistake: treating total risk as approximation error. The reasoning “the risk is positive, so the model has approximation error” is wrong. A model containing \(m\) has zero approximation error even when \(R(m)>0\), as the noise example shows. The correction is to subtract \(R(m)\): the squared-loss excess risk is exactly the squared \(L_2(P_X)\) distance from \(g\) to \(m\).
• Common mistake: assuming each added basis function must reduce error strictly. Nesting guarantees only that error does not increase. An added direction may be orthogonal to the current residual, as \(x\) was in the symmetric quadratic example, giving no reduction.