Projection Bias and Series Variance

Why Projection Separates Two Sources of Error

A series density estimator represents a density using a finite collection of orthonormal functions. Keeping more functions can represent more features of the target density, but it also means estimating more coefficients from the sample. Orthogonal projection makes these two effects precise: the part of the density outside the chosen span is approximation error, while the estimated coefficients introduce sampling variation.

The distinction is useful because it gives an exact integrated squared error identity before any asymptotic approximation. It also shows what controls the total variance: the number of included terms, the density, and the size of the basis functions over the domain.

Projection and the Series Estimator

Let \(D\) be a measurable subset of the real line, and let \(f\) be a probability density on \(D\) with \(\int_D f(x)^2\,dx<\infty\). Suppose \(X_1,\ldots,X_n\) are independent observations with density \(f\). Choose \(m\) functions \(\phi_1,\ldots,\phi_m\) that are orthonormal in \(L^2(D)\), meaning that \(\int_D \phi_j(x)\phi_k(x)\,dx\) equals 1 if \(j=k\) and 0 otherwise.

Each estimated coefficient is unbiased: \(E[\hat{\theta}_j]=\theta_j\), provided \(E[\phi_j(X_1)^2]<\infty\). The projection \(f_m\) is the best approximation to \(f\) in the chosen span under integrated squared error: its residual \(f-f_m\) is orthogonal to every function in that span. The result below uses this orthogonality to separate errors exactly.

Exact Integrated Squared Error Decomposition

Here \(\|g\|_2^2=\int_D g(x)^2\,dx\). The first identity holds for the realized estimator, not just on average. To prove it, write \(\hat{f}_m-f=(f_m-f)+(\hat{f}_m-f_m)\). The second term belongs to the span of the basis. The first is orthogonal to that span, so their integrated product is zero. Orthonormality also gives \(\|\hat{f}_m-f_m\|_2^2=\sum_{j=1}^m(\hat{\theta}_j-\theta_j)^2\). Expanding the squared norm therefore yields the first identity.

For the expected identity, take expectations of both sides. Unbiasedness gives \(E[(\hat{\theta}_j-\theta_j)^2]=\operatorname{Var}(\hat{\theta}_j)\). Since the observations are independent and identically distributed, \(\operatorname{Var}(\hat{\theta}_j)=\operatorname{Var}(\phi_j(X_1))/n\). Substitution proves the claim. \(\square\)

The first term, \(\|f-f_m\|_2^2\), is the squared projection error, sometimes called the squared approximation bias. The second is the sum of the coefficient estimation variances. No covariance terms appear in this integrated error: orthonormality makes distinct basis functions contribute separate squared coefficient errors. This does not mean the estimated coefficients are necessarily independent.

Bounding the Total Variance

The exact formula reduces the variance calculation to the second moments of the basis functions under the sampling density. A useful bound follows when the density is bounded. More generally, the same argument applies when the combined squared size of the basis functions is bounded.

To prove the bounded-density bound, use \(\operatorname{Var}(\phi_j(X_1))\leq E[\phi_j(X_1)^2]\) and sum: \(\sum_{j=1}^m E[\phi_j(X_1)^2]=\int_D f(x)\sum_{j=1}^m\phi_j(x)^2\,dx\). Since \(f(x)\leq\|f\|_\infty\) almost everywhere and the functions are orthonormal, \(\int_D\sum_{j=1}^m\phi_j(x)^2\,dx=m\). The sum of second moments is therefore at most \(m\|f\|_\infty\), and division by \(n\) proves the result. For the more general bound, the displayed integral is at most \(B_m\int_D f(x)\,dx=B_m\), because \(f\) integrates to 1. Again divide by \(n\). \(\square\)

A pointwise bound on each individual basis function can yield a looser estimate than bounding their combined squared magnitude. For a Fourier basis made of complete sine-cosine pairs, the sum of squared basis functions is constant in \(x\), so the combined bound can be particularly direct. In other settings, it may be useful to retain the exact variance formula rather than replace it with an upper bound.

What the Decomposition Tells Us

The identity applies for each fixed choice of the basis and its dimension \(m\). It does not by itself prescribe how to choose those functions or how many to include. It does establish the quantities that such a choice must balance: approximation error from projecting the density and variance from estimating the coefficients. The variance bound shows why a larger series can incur a larger sampling contribution, with a typical upper bound proportional to \(m/n\) when the density is bounded.

In applications, the exact decomposition is useful for checking derivations and for comparing candidate series spaces. The approximation term depends on how well the selected span represents the target density; the variance term depends on how the observations populate the basis functions. Both matter for integrated squared error, and neither can be inferred from the number of terms alone.

Common Mistake

A frequent error is to claim that the coefficient estimates are independent because the basis functions are orthogonal. Orthogonality is an integral statement with respect to the reference measure \(dx\); coefficient dependence is governed by expectations under the sampling density \(f\). In general, \(\operatorname{Cov}(\phi_j(X_1),\phi_k(X_1))\) need not be zero. The integrated error formula has no covariance terms for a different reason: the squared \(L^2\) norm of a sum of orthogonal functions is the sum of their squared coefficients. Do not use orthogonality to assert independence or to discard covariance in a pointwise variance calculation.