Why Projection Separates Two Sources of Error
A series estimator represents a regression function using only a selected collection of basis functions. Even if those functions are estimated perfectly, the representation may omit part of the true function. In a sample, the coefficients are also uncertain. Squared-error risk separates these two effects: the error from approximation by the selected span and the variance from estimating its coefficients.
The separation depends on orthogonality. The part of the function omitted by projection is orthogonal to every function retained in the series. Consequently, it cannot add a cross-term to the squared error of any estimate lying in that span. We establish this fact first, then derive the risk identity for estimated coefficients.
Projection onto a Finite Orthonormal System
Let \(X\) have distribution \(P_X\), and use the inner product \(\langle g,h\rangle=E[g(X)h(X)]\) and norm \(\|g\|_2^2=E[g(X)^2]\). Assume the regression function \(f(x)=E[Y\mid X=x]\) belongs to \(L^2(P_X)\). Let \(\phi_1,\ldots,\phi_K\) be measurable, square-integrable functions that are orthonormal under \(P_X\): \(E[\phi_j(X)\phi_k(X)]=1\) if \(j=k\), and \(0\) otherwise. Their span is the set of functions expressible as \(\sum_{j=1}^K c_j\phi_j\).
Orthonormality gives \(\langle f-f_K,\phi_j\rangle=0\) for every retained basis function. If the basis is part of a complete orthonormal system, Parseval's identity further expresses the approximation error as the sum of squared coefficients not retained. Completeness is not needed for the finite-dimensional projection and risk results below.
Proof. Write \(f-g=(f-f_K)+(f_K-g)\). The first term is orthogonal to every basis function and therefore to \(f_K-g\), which is a linear combination of them. Expanding the squared norm gives \(\|f-g\|_2^2=\|f-f_K\|_2^2+\|f_K-g\|_2^2+2\langle f-f_K,f_K-g\rangle\). The inner product in the last term is zero. The identity follows. Since \(\|f_K-g\|_2^2\geq 0\), \(f_K\) minimizes the distance; equality with the minimum requires \(\|f_K-g\|_2^2=0\), so \(g=f_K\) in \(L^2(P_X)\). \(\square\)
Worked Example: Computing the Omitted-Coefficient Error
Let \(X\) be uniform on three points, labeled 1, 2, and 3. Define \(\phi_1=(1,1,1)\), \(\phi_2=(\sqrt{3/2},-\sqrt{3/2},0)\), and \(\phi_3=(1/\sqrt{2},1/\sqrt{2},-\sqrt{2})\), where each vector lists its values at those points. Under the uniform distribution, these functions are orthonormal: their means of squares are 1, and the means of each pairwise product are 0.
Take \(f=\phi_1+\frac12\phi_2+\frac23\phi_3\), and retain only \(\phi_1,\phi_2\). Its projection is \(f_2=\phi_1+\frac12\phi_2\), since the inner products with the retained functions are 1 and \(1/2\). The residual is \(f-f_2=\frac23\phi_3\). Therefore, \(\|f-f_2\|_2^2=\frac49 E[\phi_3(X)^2]=\frac49\). The omitted coefficient contributes its square because the omitted basis function has unit norm.
The Squared-Error Risk Identity
Suppose \((X_i,Y_i)\), \(i=1,\ldots,n\), are independent and identically distributed, with \(E[Y\mid X]=f(X)\). Assume the moments \(E[Y^2\phi_j(X)^2]\) are finite for each \(j\). Define the empirical coefficient estimates and the series estimator by \(\hat\beta_j=n^{-1}\sum_{i=1}^nY_i\phi_j(X_i)\) and \(\hat f_K=\sum_{j=1}^K\hat\beta_j\phi_j\).
The conditional-mean assumption and finite moments ensure \(E[\hat\beta_j]=E[f(X)\phi_j(X)]=\beta_j\). The squared-error risk here is the expected integrated error \(E[\|\hat f_K-f\|_2^2]\): the norm integrates over the distribution of a new covariate \(X\), and the outer expectation is over the training data.
Proof. For every realized training sample, \(\hat f_K-f_K\) lies in the retained span, while \(f-f_K\) is orthogonal to that span. Applying the projection identity, with the sign of the span term reversed if needed, gives \(\|\hat f_K-f\|_2^2=\|f-f_K\|_2^2+\|\hat f_K-f_K\|_2^2\). Orthonormality makes the second squared norm equal to \(\sum_{j=1}^K(\hat\beta_j-\beta_j)^2\). Taking expectation yields the first risk expression because \(E[(\hat\beta_j-\beta_j)^2]=\operatorname{Var}(\hat\beta_j)\), by unbiasedness. Finally, each \(\hat\beta_j\) is an average of \(n\) independent copies of \(Y\phi_j(X)\), so \(\operatorname{Var}(\hat\beta_j)=\operatorname{Var}(Y\phi_j(X))/n\). All sums are finite, and the assumed second moments make these expectations finite. \(\square\)
The variance term is a sum of coefficient variances, not a sum of covariances. Even if estimated coefficients are dependent, orthonormality makes the integrated squared norm of their series difference equal to the sum of their squared errors. The theorem also shows why the variance is not always simply \(K\sigma^2/n\): with random covariates, each \(Y\phi_j(X)\) has its own sampling variance.
Worked Example: Fixed-Design Risk with Two Coefficients
Consider four fixed design points with empirical inner product \(\langle u,v\rangle_4=\frac14\sum_{i=1}^4u_iv_i\). Let the two basis vectors be \(\phi_1=(1,1,1,1)\) and \(\phi_2=(1,1,-1,-1)\). Their empirical squared norms are both 1, and their empirical inner product is \((1+1-1-1)/4=0\).
Let the true mean vector be \(f=2\phi_1+\phi_2+r\), where \(r=(1,-1,-1,1)/2\). The empirical inner products of \(r\) with \(\phi_1\) and \(\phi_2\) are both zero, so the projection is \(f_2=2\phi_1+\phi_2\). The omitted error is \(\|r\|_4^2=(1/4)(4\cdot 1/4)=1/4\).
Observe \(Y_i=f_i+\epsilon_i\), with independent \(\epsilon_i\sim N(0,1)\). The coefficient estimates are \(\hat\beta_j=\frac14\sum_{i=1}^4\phi_j(i)Y_i\). Each has variance \(\frac1{16}\sum_i\phi_j(i)^2\operatorname{Var}(\epsilon_i)=4/16=1/4\). Thus the variance contribution is \(1/4+1/4=1/2\), and the expected integrated risk is \(1/4+1/2=3/4\).
Worked Example: Random-Design Coefficient Variance
Use the three-point uniform design and basis \(\phi_1,\phi_2,\phi_3\) from the earlier example, retaining \(\phi_1,\phi_2\). Set \(a=2/\sqrt{45}\), \(b=\sqrt{242/945}\), and \(f=b\phi_2+a\phi_3\). Observe \(Y=f(X)+\epsilon\), where \(\epsilon\sim N(0,\sigma^2)\) is independent of \(X\). The retained coefficients are \(0\) and \(b\), and the approximation error is \(a^2=4/45\).
For the sum of coefficient variances, let \(w=\phi_1^2+\phi_2^2\). Its values at the three design points are \(5/2,5/2,1\), so \(E[w]=2\). The cross-product \(E[w\phi_2\phi_3]\) is zero: its first two terms cancel and its third term is zero. Also \(E[w\phi_2^2]=5/2\) and \(E[w\phi_3^2]=3/2\). Since the noise is independent with mean zero, \(\sum_{j=1}^2\operatorname{Var}(Y\phi_j(X))=E[Y^2w]-\sum_{j=1}^2(E[Y\phi_j(X)])^2=\frac52b^2+\frac32a^2+2\sigma^2-b^2\).
The signal part simplifies to \(\frac32b^2+\frac32a^2\). Substituting \(b^2=242/945\) and \(a^2=4/45\) gives \(\frac32\frac{242}{945}+\frac32\frac4{45}=\frac{121}{315}+\frac{42}{315}=\frac{163}{315}=\frac47? \) More directly, \(\frac{163}{315}=\frac4{45}+\frac37\), since \(\frac4{45}=\frac{28}{315}\) and \(\frac37=\frac{135}{315}\). Therefore, for sample size \(n\), the exact risk is \(E[\|\hat f_2-f\|_2^2]=\frac4{45}+\frac1n\left(\frac4{45}+\frac37+2\sigma^2\right)\). The variance constants sum to \(163/315\), or approximately \(0.5174603\), not \(0.5333333\).
Interpretation and a Common Pitfall
The identity is useful because it distinguishes what a chosen span can represent from how accurately its coefficients can be learned. The approximation term is determined by \(f\), the design distribution, and the retained basis. The variance term is determined by the sampling distribution and sample size. In particular, the projection error is not sampling variance: it remains even if the coefficients were known exactly.
- Pitfall: adding a cross-term between approximation error and coefficient error. A generic expansion of a squared error does contain a cross-term. Here it is zero because the projection residual is orthogonal to every function in the retained span, for every realized coefficient vector.
- Pitfall: treating all coefficient variances as identical. Under random design, \(\operatorname{Var}(Y\phi_j(X))\) can differ across basis functions. The shortcut \(K\sigma^2/n\) requires additional conditions, such as a fixed orthonormal design and independent homoskedastic errors with variance \(\sigma^2\).
- Pitfall: counting coefficient covariances in integrated squared error. Orthonormality reduces the squared norm of a linear combination to the sum of squared coefficients. Covariances between coefficient estimates do not enter this particular integrated risk identity.
Check Your Understanding
Use the projection and risk identities to answer these questions.
- Why is the residual \(f-f_K\) orthogonal to every function in the retained span?
- If the retained coefficients were known without sampling error, which term in the risk identity would remain?
- For a complete orthonormal basis, how can the approximation error be written in terms of omitted coefficients?
- Why does the risk identity not include covariance terms between different estimated coefficients?
- Under what additional design and noise conditions can the variance contribution simplify to \(K\sigma^2/n\)?