Truncation Risk and Series Dimension Choice

Why the Number of Series Terms Matters

A series estimator represents a regression function using an ordered collection of basis functions and keeps only the first $D$ terms. Choosing $D$ is a balance: a small dimension can omit important structure, while a large dimension estimates more coefficients and accumulates more noise. The risk calculation makes this trade-off precise and gives an oracle choice: the dimension that would minimize risk if the underlying series coefficients were known.

We work in a fixed-design setting so the orthogonality and risk statements are exact. The loss is average squared error at the observed design points. This setup isolates truncation from other issues, such as whether a random-design Gram matrix is close to its population counterpart.

Model and Truncated Estimator

Suppose the responses satisfy $Y_i=f_i+\varepsilon_i$, for $i=1,\ldots,n$, where the design points are fixed, $f_i=f(x_i)$, and the errors are independent with $E[\varepsilon_i]=0$ and $\operatorname{Var}(\varepsilon_i)=\sigma^2<\infty$. No Gaussian assumption is needed. Let $\phi_1,\ldots,\phi_n$ be a complete orthonormal basis for vectors on these $n$ design points, using the empirical inner product $\langle u,v\rangle_n=n^{-1}\sum_{i=1}^n u_i v_i$. Thus $n^{-1}\sum_i\phi_j(x_i)\phi_k(x_i)$ equals 1 if $j=k$ and 0 otherwise.

Write $\theta_j=\langle f,\phi_j\rangle_n$ for the coefficients of the regression vector. Completeness gives $f_i=\sum_{j=1}^n\theta_j\phi_j(x_i)$. The empirical coefficient estimate is $\hat\theta_j=n^{-1}\sum_i\phi_j(x_i)Y_i$, and the estimator truncated at $D$ is $\hat f_D(x_i)=\sum_{j=1}^D\hat\theta_j\phi_j(x_i)$, for $0\leq D\leq n$.

Definition: The truncation risk at dimension $D$ is $R(D)=E[\|\hat f_D-f\|_n^2]$, where $\|g\|_n^2=n^{-1}\sum_{i=1}^n g_i^2$. An oracle dimension is any value of $D$ in the allowed range that minimizes this risk, assuming the coefficients and noise variance that determine the risk are known.

Exact Risk Decomposition

Orthonormality turns integrated squared error into a sum of squared coefficient errors. The estimated coefficient has expectation $\theta_j$. Its variance follows directly from independence and equal error variances: $\hat\theta_j-\theta_j=n^{-1}\sum_i\phi_j(x_i)\varepsilon_i$, so its variance is $\sigma^2/n$. These facts give the exact risk, not just an asymptotic approximation.

Theorem: Under the fixed-design model and orthonormality conditions above, for every $D\in\{0,\ldots,n\}$, $$ R(D)=\sum_{j=D+1}^{n}\theta_j^2+\frac{\sigma^2D}{n}. $$ The first term is squared truncation bias; the second is estimation variance.

Proof. Orthonormality implies the Parseval identity on the $n$ design points: $\|\hat f_D-f\|_n^2=\sum_{j=1}^D(\hat\theta_j-\theta_j)^2+\sum_{j=D+1}^n\theta_j^2$. The second sum is deterministic. For $j\leq D$, unbiasedness gives $E[(\hat\theta_j-\theta_j)^2]=\operatorname{Var}(\hat\theta_j)$. Independence of the errors and empirical orthonormality yield

\operatorname{Var}(\hat\theta_j) =\frac{1}{n^2}\sum_{i=1}^n\phi_j(x_i)^2\sigma^2 =\frac{\sigma^2}{n}.

Taking expectations of the Parseval decomposition and summing the $D$ coefficient variances proves the formula. No independence between different estimated coefficients is required: Parseval expresses the loss as a sum of their squared errors, without cross-products. $\square$

Worked Example: Exact Risk and the Oracle Dimension

Suppose $n=100$, $\sigma^2=4$, and the first four squared coefficients are $0.30, 0.10, 0.06, 0.01$; all later coefficients are zero. Then each included term contributes variance $\sigma^2/n=0.04$. Compute the risk for dimensions 2, 3, and 4.

At $D=2$, the omitted squared coefficients total $0.06+0.01=0.07$, and the variance is $2(0.04)=0.08$. Thus $R(2)=0.07+0.08=0.15$.

At $D=3$, the omitted squared coefficient total is $0.01$, and the variance is $3(0.04)=0.12$. Thus $R(3)=0.01+0.12=0.13$.

At $D=4$, no nonzero coefficient is omitted, and the variance is $4(0.04)=0.16$. Thus $R(4)=0.16$. Among these dimensions, $D=3$ has the smallest risk. The first three coefficient squares exceed the variance cost $0.04$ per added term, but the fourth does not.

Choosing a Dimension by Marginal Risk

The exact formula gives a simple comparison between neighboring dimensions. Increasing $D$ by one removes the squared coefficient $\theta_{D+1}^2$ from the truncation error and adds variance $\sigma^2/n$. Hence the decision depends on whether that coefficient's squared size compensates for its estimation cost.

Result: For $0\leq D<n$, $$ R(D+1)-R(D)=\frac{\sigma^2}{n}-\theta_{D+1}^2. $$ Adding term $D+1$ decreases risk exactly when $\theta_{D+1}^2>\sigma^2/n$, increases it when $\theta_{D+1}^2<\sigma^2/n$, and leaves it unchanged at equality.

Derivation. Subtract the risk at $D$ from the risk at $D+1$. Every tail term cancels except $\theta_{D+1}^2$, which is removed, while the variance term increases by one increment:

R(D+1)-R(D) =\left(\sum_{j=D+2}^n\theta_j^2+\frac{\sigma^2(D+1)}{n}\right) -\left(\sum_{j=D+1}^n\theta_j^2+\frac{\sigma^2D}{n}\right) =\frac{\sigma^2}{n}-\theta_{D+1}^2.

This comparison proves the stated rule. It does not, without additional conditions on the sequence of coefficient sizes, imply that the risk is globally minimized at the first point where an added coefficient fails the threshold. In full generality, compare the risk over all allowed dimensions or verify that the successive differences change sign only once. $\square$

Worked Example: A Continuous Approximation to the Oracle Choice

Suppose the squared truncation error is approximated by $D^{-2}$, and take $n=1024$ and $\sigma^2=1$. The resulting approximate risk for positive $D$ is $R_{\mathrm{approx}}(D)=D^{-2}+D/1024$. Treating $D$ as continuous gives a useful guide; the actual choice must be an integer.

Differentiate the surrogate risk: $R_{\mathrm{approx}}'(D)=-2D^{-3}+1/1024$. Setting the derivative equal to zero gives $2/D^3=1/1024$, hence $D^3=2(1024)=2048$. The continuous minimizer is therefore $D=2048^{1/3}$, approximately $12.70$.

Compare the neighboring integer choices. At $D=12$, $R_{\mathrm{approx}}(12)=1/144+12/1024$, approximately $0.018663$. At $D=13$, $R_{\mathrm{approx}}(13)=1/169+13/1024$, approximately $0.018612$. Thus $D=13$ is better of these two integer choices. Since the derivative changes from negative to positive at the continuous minimizer, the surrogate is decreasing before $12.70$ and increasing after it; the nearest competing integers are 12 and 13, so this comparison identifies the minimizing integer.

This is an optimization of an approximate risk curve, not an exact risk calculation for specified coefficients. The calculation illustrates the balance and the rounding step; the exact oracle dimension in a particular model is determined by its actual coefficient tail.

How Smoothness Determines the Scale of the Choice

A common approximation assumption is that the squared tail after $D$ terms is bounded by $A D^{-2s}$, where $A>0$ and $s>0$ describes the rate at which the series coefficients decay. Using this bound with the exact variance term gives the risk bound $A D^{-2s}+\sigma^2D/n$. Balancing these two terms provides a dimension scale; it is not necessary to know the exact coefficients to understand this rate calculation.

For a continuous positive dimension, differentiate this bound. Its derivative is $-2sA D^{-2s-1}+\sigma^2/n$. Equating it to zero yields $D^{2s+1}=2sAn/\sigma^2$, or

D_{\mathrm{bal}}=\left(\frac{2sAn}{\sigma^2}\right)^{1/(2s+1)}.

At this scale, the two risk contributions have the same order in $n$: each is proportional to $n^{-2s/(2s+1)}$, with constants depending on $A$, $s$, and $\sigma^2$. The dimension must in practice be an integer between 0 and $n$, so one rounds a suitable scale and respects those bounds. The rate describes how the balance changes with sample size; it does not supply an exact oracle dimension unless the risk assumptions are exact.

Worked Example: Balancing a Fourth-Order Tail Bound

Let the squared tail be bounded by $D^{-4}$, so $A=1$ and $s=2$, and suppose $\sigma^2=1$ and $n=1000$. The continuous balance is $D_{\mathrm{bal}}=(4\cdot1000)^{1/5}=4000^{1/5}$, approximately $5.26$.

For the surrogate risk $D^{-4}+D/1000$, check the adjacent integers. At $D=5$, the risk is $1/625+5/1000=0.0016+0.005=0.0066$. At $D=6$, it is $1/1296+6/1000$, approximately $0.000772+0.006=0.006772$. Thus $D=5$ is lower of these candidates. The derivative of the surrogate is $-4D^{-5}+1/1000$, which vanishes at $4000^{1/5}$; it is negative below and positive above that point. Therefore 5 is the minimizing integer, not merely a rounded guess.

Interpretation and a Common Pitfall

The oracle calculation clarifies why truncation is a statistical decision rather than just a computational one. Each term reduces approximation error by an amount determined by its coefficient, while increasing variance by a fixed amount in this orthonormal, homoskedastic setting. The best dimension depends both on the signal's coefficient decay and on the noise level relative to sample size.

Pitfall: minimizing variance alone. Since variance increases with $D$, choosing $D=0$ would minimize that part of risk. But it can leave a large tail of omitted signal. The correction is to minimize total risk, including truncation error.
Pitfall: treating the oracle rule as directly observable. The threshold compares unknown $\theta_j^2$ with $\sigma^2/n$. It characterizes the risk-minimizing decision when those quantities are known; it does not itself provide a data-based selection procedure.
Pitfall: assuming the formula holds under any design and basis. The exact variance increment here uses empirical orthonormality and independent errors with common variance. Different weighting, heteroskedastic errors, or a nonorthogonal empirical basis changes the calculation.

The main practical lesson is to keep the target loss and the orthogonality assumptions visible. Under the stated conditions, the bias–variance decomposition and marginal comparison are exact. Under other conditions, they may remain useful guides, but their variance term must be derived for that setting rather than borrowed unchanged.

Check Your Understanding

Use the assumptions and risk formula developed above.

Why does the variance contribution increase by $\sigma^2/n$ when the truncation dimension increases by one?
If $\theta_{D+1}^2=\sigma^2/n$, what is the difference between the risks at dimensions $D+1$ and $D$?
For $n=200$ and $\sigma^2=2$, what variance cost is added by one more term?
In the bound $A D^{-2s}+\sigma^2D/n$, how does the balanced dimension scale with $n$, apart from constants?
Why does the marginal inclusion rule not automatically identify an observable, data-based dimension choice?