Tensor-Product Series for Anisotropic Smoothness

Why Rectangular Truncation?

A function of several predictors need not vary equally in every direction. For example, a regression surface might change rapidly with treatment dose but gradually with age. A series estimator that retains the same frequency range in every coordinate can spend many coefficients resolving a direction that is already smooth, while undersmoothing another direction. Rectangular truncation addresses this by assigning a separate frequency cutoff to each coordinate.

We develop the construction for periodic regression on $[0,1]^d$ with uniform random design. The Fourier basis makes both the approximation argument and the variance calculation explicit. The conclusions illustrate the central mechanism: rougher coordinates receive larger cutoffs, while the product of all cutoffs determines the number of estimated coefficients.

Tensor-Product Basis and Anisotropic Smoothness

In one coordinate, use the real trigonometric functions $1$, $\sqrt{2}\cos(2\pi m x)$, and $\sqrt{2}\sin(2\pi m x)$, for integers $m\geq 1$. They form an orthonormal basis of $L^2([0,1])$ for periodic functions. In $d$ coordinates, take products of these functions, one factor per coordinate. Write a basis function as $\phi_k$, where $k=(k_1,\ldots,k_d)$ records the constant, cosine, or sine frequency in each coordinate. Let $\nu_j(k)$ be the nonnegative integer frequency in coordinate $j$, with frequency zero for a constant factor. The products are orthonormal in $L^2([0,1]^d)$, with respect to Lebesgue measure.

For $f\in L^2([0,1]^d)$, its coefficient at $k$ is $\theta_k=\int_{[0,1]^d} f(x)\phi_k(x)\,dx$. The following ellipsoid expresses smoothness separately in each direction.

Definition: For positive smoothness indices $s_1,\ldots,s_d$ and radius $R>0$, the anisotropic Fourier ellipsoid consists of functions whose coefficients satisfy $$ \sum_k \left(1+\sum_{j=1}^d \nu_j(k)^{2s_j}\right)\theta_k^2\leq R^2. $$ The index $s_j$ measures smoothness in coordinate $j$; larger $s_j$ imposes stronger decay of coefficients with frequency in that coordinate.

For positive integer cutoffs $J_1,\ldots,J_d$, the rectangular index set retains every tensor basis function whose coordinate frequencies satisfy $\nu_j(k)\leq J_j$ for every $j$. It contains $M=\prod_{j=1}^d(2J_j+1)$ basis functions: in each coordinate there is one constant and two functions at each positive frequency.

Worked Example: Counting Coefficients in a Rectangle

For two predictors, choose $J_1=2$ and $J_2=1$. The first coordinate contributes the constant and two functions at each of frequencies 1 and 2, for $1+2(2)=5$ basis functions. The second contributes the constant and two functions at frequency 1, for $1+2(1)=3$. Their tensor products give $$ M=(2\cdot 2+1)(2\cdot 1+1)=5\cdot 3=15. $$ Thus the estimator has 15 coefficients, including products such as a frequency-2 cosine in the first coordinate times a frequency-1 sine in the second. It is a rectangle of frequencies, not a list of 15 one-dimensional functions.

Approximation Error for Rectangular Cutoffs

Let $f_J$ be the orthogonal projection of $f$ onto the retained rectangle: $f_J=\sum_{\nu_j(k)\leq J_j\ \forall j}\theta_k\phi_k$. The next result bounds the energy discarded outside that rectangle.

Theorem: If $f$ belongs to the anisotropic Fourier ellipsoid, then for all positive integer cutoffs, $$ \|f-f_J\|_2^2\leq R^2\sum_{j=1}^d J_j^{-2s_j}. $$

Proof. By orthonormality and Parseval's identity, the squared error is the sum of $\theta_k^2$ over indices outside the rectangle. Any such index has $\nu_j(k)>J_j$ for at least one coordinate $j$. Therefore the sum over indices outside the rectangle is at most the sum, over coordinates, of the coefficient sums with $\nu_j(k)>J_j$. For each such term, $\nu_j(k)^{2s_j}>J_j^{2s_j}$, so $$ \begin{aligned} \|f-f_J\|_2^2 &\leq \sum_{j=1}^d \sum_{\nu_j(k)>J_j}\theta_k^2\\ &\leq \sum_{j=1}^d J_j^{-2s_j} \sum_k \nu_j(k)^{2s_j}\theta_k^2\\ &\leq R^2\sum_{j=1}^d J_j^{-2s_j}. \end{aligned} $$ The last inequality follows because each coordinate-weighted coefficient sum is at most the ellipsoid sum, which is at most $R^2$. This proves the bound. $\square$

The sum in this bound comes from covering the discarded region by the union of the coordinate-wise tails. It is an upper bound; it need not equal the actual error, since an index outside the rectangle can be counted in more than one coordinate tail.

Worked Example: Evaluating an Approximation Bound

Suppose $d=2$, $s_1=1$, $s_2=2$, and choose $J_1=14$, $J_2=4$. The theorem gives $$ \|f-f_J\|_2^2 \leq R^2\left(14^{-2}+4^{-4}\right) =R^2\left(\frac{1}{196}+\frac{1}{256}\right). $$ Numerically, $1/196\approx0.00510204$ and $1/256=0.00390625$, so the sum is $0.00900829\ldots$, or approximately $0.00901$. Thus the bound is approximately $0.00901R^2$. The calculation shows how a relatively modest cutoff in the smoother second coordinate can contribute as little as a larger cutoff in the rougher first coordinate.

Empirical Coefficients and a Variance Bound

Observe independent and identically distributed pairs $(X_i,Y_i)$, $i=1,\ldots,n$, from the regression model $X_i\sim\operatorname{Unif}([0,1]^d)$, $Y_i=f(X_i)+\varepsilon_i$. Assume $E[\varepsilon_i\mid X_i]=0$, $E[\varepsilon_i^2\mid X_i]\leq\sigma^2$ almost surely, and $|f(x)|\leq B$ almost everywhere for a finite constant $B$. These conditions ensure a finite second moment and unbiased empirical coefficients.

For each retained basis function, define $\hat\theta_k=n^{-1}\sum_{i=1}^nY_i\phi_k(X_i)$, and set $\hat f_J=\sum_{\nu_j(k)\leq J_j\ \forall j}\hat\theta_k\phi_k$.

Theorem: Under the stated regression assumptions, the integrated mean squared error obeys $$ E\|\hat f_J-f\|_2^2 \leq R^2\sum_{j=1}^d J_j^{-2s_j} +\frac{(B^2+\sigma^2)}{n}\prod_{j=1}^d(2J_j+1), $$ whenever $f$ also belongs to the anisotropic Fourier ellipsoid.

Proof. Uniform design and orthonormality give $E[Y_i\phi_k(X_i)]=\int f(x)\phi_k(x)\,dx=\theta_k$, so every retained coefficient estimate is unbiased. Orthogonality splits the squared error into the deterministic omitted-coefficient sum and the squared errors of retained coefficients. Taking expectations yields $$ E\|\hat f_J-f\|_2^2 =\|f-f_J\|_2^2+\sum_{k\ \mathrm{retained}}\operatorname{Var}(\hat\theta_k). $$ For each $k$, independence across observations gives $\operatorname{Var}(\hat\theta_k)\leq n^{-1}E[Y^2\phi_k(X)^2]$. Summing and interchanging a finite sum and expectation, $$ \sum_{k\ \mathrm{retained}}\operatorname{Var}(\hat\theta_k) \leq \frac{1}{n}E\left[Y^2\sum_{k\ \mathrm{retained}}\phi_k(X)^2\right]. $$ In one coordinate, the squared basis functions through frequency $J_j$ sum to $1+2\sum_{m=1}^{J_j}(\cos^2(2\pi m x_j)+\sin^2(2\pi m x_j))=1+2J_j$. The tensor-product sum is the product of these coordinate sums, so it is exactly $\prod_j(2J_j+1)$, independent of $X$. Also, $E[Y^2\mid X]=f(X)^2+E[\varepsilon^2\mid X]\leq B^2+\sigma^2$. Consequently the retained-coefficient variance sum is at most $(B^2+\sigma^2)\prod_j(2J_j+1)/n$. Applying the approximation theorem to the omitted-coefficient term proves the displayed inequality. $\square$

Worked Example: A Finite-Sample Risk Bound

Use the two-dimensional cutoffs $J_1=2$, $J_2=1$, and suppose $f$ is in an ellipsoid with $s_1=1$, $s_2=2$. The preceding results give $$ E\|\hat f_J-f\|_2^2 \leq R^2\left(2^{-2}+1^{-4}\right) +\frac{15(B^2+\sigma^2)}{n} =\frac{5}{4}R^2+\frac{15(B^2+\sigma^2)}{n}. $$ The first term is the approximation bound, since $1/4+1=5/4$. The second uses the 15 retained coefficients calculated earlier. Increasing the cutoffs can reduce the first term, but necessarily increases the second through the number of estimated coefficients.

Choosing Direction-Specific Cutoffs

The risk bound suggests balancing the coordinate-wise approximation terms against the coefficient cost. Write $S=\sum_{j=1}^d 1/s_j$. To make each $J_j^{-2s_j}$ of order $n^{-2/(2+S)}$, choose cutoffs on the scale $J_j\asymp n^{1/[s_j(2+S)]}$. Then their product is of order $n^{S/(2+S)}$, and the variance term is of order $n^{-2/(2+S)}$. The following example checks the exponents directly.

Worked Example: Balancing Two Different Smoothness Levels

Let $d=2$, $s_1=1$, and $s_2=2$. Then $S=1+1/2=3/2$, so $2+S=7/2$. The balancing choices are $J_1\asymp n^{2/7}$ and $J_2\asymp n^{1/7}$. Their product is of order $n^{3/7}$, making the variance order $n^{3/7}/n=n^{-4/7}$. The approximation terms are $J_1^{-2}\asymp n^{-4/7}$ and $J_2^{-4}\asymp n^{-4/7}$ as well. Thus the risk bound has order $n^{-4/7}$, up to constants. The rougher first coordinate receives the larger frequency cutoff, as it should.

More generally, the same calculation gives the rate $n^{-2/(2+S)}$ for this bound when the smoothness indices are fixed and the integer cutoffs are chosen at the stated scales. The factor $2J_j+1$ is comparable to $J_j$ for growing cutoffs, so it does not change the rate. This is a rate calculation for the specified estimator and assumptions, not a claim that every regression model or every smoother basis has the same constants.

Common Pitfall and Practical Meaning

A common mistake is to impose one cutoff $J$ in every coordinate merely because the basis is a tensor product. That choice gives variance proportional to $J^d/n$, while the approximation terms become $J^{-2s_j}$ in each direction. It forces a single scale to serve coordinates with possibly very different coefficient decay. Rectangular cutoffs preserve the product structure of the variance cost but allow each smoothness direction to be handled separately.

The ellipsoid assumption is substantive: it specifies how coefficient energy decays, and the risk calculation uses that decay explicitly. Likewise, the random-design variance bound uses uniform design, bounded $f$, and the stated conditional noise bound. If the design is nonuniform or the basis has a different pointwise squared-sum bound, that part of the argument must be changed rather than carried over unchanged.

Key takeaway: In tensor-product series estimation, coordinate-wise smoothness leads naturally to coordinate-wise frequency cutoffs. The approximation cost is controlled by $\sum_j J_j^{-2s_j}$, while the variance cost is controlled by the rectangle size $\prod_j(2J_j+1)/n$.

Check Your Understanding

Use the definitions and bounds above to answer the following questions.

For $d=3$ and cutoffs $J_1=1$, $J_2=2$, $J_3=3$, how many tensor-product coefficients are retained?
Why does the approximation proof use a sum of coordinate-wise tail bounds rather than treating the discarded region as just one coordinate tail?
In the empirical risk bound, where does the factor $\prod_j(2J_j+1)$ come from?
If one smoothness index increases while the others stay fixed, how does the balancing exponent for that coordinate's cutoff change?
Which assumptions in the variance proof would need reconsideration under a nonuniform random design?