Series Risk Under Heteroskedastic Noise

Why the Noise Profile Matters

A series estimator approximates a regression function with a finite collection of basis functions. Its prediction risk reflects both the error from truncating that approximation and the variability of the estimated coefficients. When the regression errors have constant variance, the noise contribution is relatively uniform across the design. When their variance depends on the design point, some parts of the domain can contribute more uncertainty than others. A basis term concentrated in a high-variance region can therefore cost more than one with the same scale concentrated in a low-variance region.

The key calculation is not just the variance of each coefficient separately. In a general basis, coefficient errors can be correlated, and integrated prediction risk combines their covariance using the geometry of the basis. The formulas below make both features explicit and give a direct way to compare truncation choices.

Setup and Definitions

Let \(x_1,\ldots,x_n\) be fixed design points, and let \(p_K(x)=(\phi_1(x),\ldots,\phi_K(x))^\top\) be a vector of \(K\) basis functions. Write \(P\) for the \(n\times K\) matrix whose \(i\)th row is \(p_K(x_i)^\top\). Suppose \(P\) has full column rank and \(Y_i=m(x_i)+\varepsilon_i\), where the errors are independent, \(E[\varepsilon_i\mid x_1,\ldots,x_n]=0\), and \(\operatorname{Var}(\varepsilon_i\mid x_1,\ldots,x_n)=\sigma^2(x_i)\).

The coefficient \(\beta_{K,n}\) is the least-squares projection of the vector of true regression values onto the columns of \(P\). It need not make the approximation error zero. The prediction risk on the observed design is \(R_n=E[n^{-1}\sum_{i=1}^n\{\hat m_K(x_i)-m(x_i)\}^2\mid x_1,\ldots,x_n]\). This risk measures average squared prediction error at the design points; it is not an average over a different target distribution.

Exact Coefficient Variance and Fixed-Design Risk

To prove the covariance formula, write \(Y=m_n+\varepsilon\). Since \(E[\varepsilon\mid x_1,\ldots,x_n]=0\), the random part of \(\hat\beta_K\) is \((P^\top P)^{-1}P^\top\varepsilon\). The conditional covariance of \(\varepsilon\) is the diagonal matrix \(D=\operatorname{diag}(\sigma^2(x_1),\ldots,\sigma^2(x_n))\). Independence gives this diagonal covariance, so

For the risk identity, decompose the fitted error into its deterministic approximation part and its centered estimation error: \(\hat m_K(x_i)-m(x_i)=p_K(x_i)^\top(\hat\beta_K-\beta_{K,n})+\{p_K(x_i)^\top\beta_{K,n}-m(x_i)\}\). The empirical projection residual vector is orthogonal to every column of \(P\), because \(P^\top(m_n-P\beta_{K,n})=0\). Consequently the cross term in the average squared error is zero for every coefficient perturbation. Taking conditional expectations leaves the squared projection residual plus \(E[(\hat\beta_K-\beta_{K,n})^\top Q_n(\hat\beta_K-\beta_{K,n})\mid x_1,\ldots,x_n]\). For a centered random vector \(Z\) and fixed matrix \(A\), \(E[Z^\top A Z]=\operatorname{tr}(A\operatorname{Cov}(Z))\). Applying this identity and the covariance formula yields \(\operatorname{tr}(Q_n n^{-1}Q_n^{-1}\Omega_n Q_n^{-1})=n^{-1}\operatorname{tr}(Q_n^{-1}\Omega_n)\). This proves both claims. \(\square\)

The first term is the empirical truncation error. The second is the integrated coefficient-variance contribution, measured in the prediction geometry \(Q_n\). The variance matrix \(\Omega_n\) contains the design-dependent noise levels. If the errors are homoskedastic with variance \(\sigma^2\), then \(\Omega_n=\sigma^2Q_n\), and the variance contribution reduces to \(\sigma^2K/n\). That simplification generally does not hold under heteroskedasticity.

Random-Design Interpretation

For random design, let \(X\) have distribution \(F\), assume \(E[\varepsilon\mid X]=0\) and \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2(X)\), and define \(Q_K=E[p_K(X)p_K(X)^\top]\) and \(\Omega_K=E[\sigma^2(X)p_K(X)p_K(X)^\top]\). Suppose \(K\) is fixed, \(Q_K\) is positive definite, the required second moments are finite, and the observations are independent and identically distributed. In a correctly specified \(K\)-term model, the usual least-squares sandwich covariance has leading term \(n^{-1}Q_K^{-1}\Omega_KQ_K^{-1}\). Its integrated prediction-variance contribution, measured under the same design distribution \(F\), is therefore \(n^{-1}\operatorname{tr}(Q_K^{-1}\Omega_K)\).

This expression also describes the noise component of the leading risk calculation when considering a series approximation. In that case, random variation in the approximation residual can add to the sandwich “meat”: for the population projection residual \(r_K(X)=m(X)-p_K(X)^\top\beta_K\), the relevant matrix is \(E[(\sigma^2(X)+r_K(X)^2)p_K(X)p_K(X)^\top]\). The extra term records that the response differs from its projection not only through measurement noise but also through the residual left by truncation. Thus a noise-only variance calculation should not be mistaken for the complete random-design covariance under misspecification.

How Heteroskedasticity Changes Truncation

For a nested, orthonormal series under the target design distribution, write the population regression function’s coefficients as \(\beta_j\). At a fixed design, the analogous statements hold for an empirically orthonormal basis. Define \(v_j=E[\sigma^2(X)\phi_j(X)^2]\), the noise contribution for term \(j\). The projection bias from truncating after \(K\) terms is \(\sum_{j>K}\beta_j^2\), and the noise contribution is \(n^{-1}\sum_{j\le K}v_j\). These expressions give a simple risk comparison.

To verify the formula, subtract the two risks. The bias sum for \(K+1\) omits one additional term, so it changes by \(-\beta_{K+1}^2\). The variance sum includes one additional contribution, changing by \(v_{K+1}/n\). Adding these changes gives the stated difference; it is negative precisely when the coefficient’s squared signal exceeds its variance cost. \(\square\)

This calculation explains what “heteroskedasticity changes optimal truncation” means. It changes the \(v_j\), and so it can change whether the next term reduces risk. In a nonorthogonal basis, individual terms need not have this separate risk-increment interpretation: the full expression \(n^{-1}\operatorname{tr}(Q_K^{-1}\Omega_K)\) accounts for the geometry and covariance jointly.

Common Pitfall and Practical Use

A common mistake is to replace every \(\sigma^2(x_i)\) by the average noise variance and then use the homoskedastic formula \(\sigma^2K/n\). That shortcut is valid when the basis geometry makes the variance matrix proportional to the Gram matrix, as under constant variance. Under heteroskedasticity, the correct term is \(n^{-1}\operatorname{tr}(Q_n^{-1}\Omega_n)\). Even when the average variance is known, the placement of high-variance observations relative to the basis functions matters.

Another pitfall is to report only the diagonal coefficient variances when the goal is integrated prediction risk. Correlations between coefficient estimates may matter through the matrix expression, especially when the basis is not orthogonal in the target prediction metric. For an orthonormal basis, the risk trace simplifies to the sum of termwise variance costs, but the coefficients can still be correlated.

In practice, the risk decomposition helps separate two decisions: how well a series basis approximates the regression function, and how much noise is paid to estimate its coefficients. Under heteroskedasticity, a term with a modest approximation benefit may be unattractive if it places substantial weight where errors are especially variable. Truncation should therefore reflect both the signal coefficients and the design-weighted variance profile, not just the nominal number of basis functions.