Worked Risk Comparison for Smooth and Jump Signals

Why Compare Fourier and Wavelet Risks?

A global Fourier basis and a spatially localized wavelet basis can represent the same signal in very different ways. For a smooth signal, many Fourier coefficients may be small and decay regularly. A jump, by contrast, produces Fourier coefficients that decay slowly across frequencies, while wavelets can concentrate much of its representation near the jump. These differences affect both approximation error and the way coefficient estimates respond to noise.

We will make the comparison through explicit risk calculations. For Fourier truncation, the main terms are squared bias from omitted coefficients and variance from estimated coefficients that are retained. For wavelet thresholding, the decision to retain or discard each coefficient depends on its noisy estimate. We will calculate that risk for a Gaussian coefficient estimate, then work through finite-resolution examples. The finite-resolution qualification matters: retaining infinitely many noisy coefficients with nonvanishing variance need not produce a square-integrable estimate.

Risk for a Finite Orthonormal Expansion

Let $\{\varphi_m\}_{m\geq 0}$ be a complete orthonormal basis of $L^2[0,1]$, and let $f=\sum_{m\geq 0}\theta_m\varphi_m$, where $\theta_m=\langle f,\varphi_m\rangle$. For a finite index set $S$, suppose we estimate the retained coefficients by $\hat\theta_m=\theta_m+\varepsilon_m$, with $E[\varepsilon_m]=0$ and $\operatorname{Var}(\varepsilon_m)=v$. Define $\hat f_S=\sum_{m\in S}\hat\theta_m\varphi_m$.

Theorem: For finite $S$, under the conditions above, $$ E\|\hat f_S-f\|_2^2 = \sum_{m\notin S}\theta_m^2+|S|v. $$ No independence assumption on the coefficient errors is needed for this identity.

To prove the formula, subtract the two basis expansions:

\hat f_S-f = \sum_{m\in S}\varepsilon_m\varphi_m - \sum_{m\notin S}\theta_m\varphi_m.

The first sum is finite. The second converges in $L^2$, since $f\in L^2[0,1]$. Orthonormality makes the retained and omitted terms orthogonal, so Parseval's identity gives

\|\hat f_S-f\|_2^2 = \sum_{m\in S}\varepsilon_m^2+\sum_{m\notin S}\theta_m^2.

Taking expectations is valid: the first sum has finitely many terms, each with finite second moment, and the second is a deterministic finite-or-convergent nonnegative series. Thus its expectation is $\sum_{m\in S}v+\sum_{m\notin S}\theta_m^2$, as claimed. $\square$

In a regression setting with coefficient estimates having variance $\sigma^2/n$, this identity separates risk into approximation error and a variance cost of $|S|\sigma^2/n$. It is an exact finite-set calculation, not a claim that an arbitrary infinite sequence of noisy coefficients defines an $L^2$ function.

Worked Fourier Comparisons

Worked Example: Truncating a Smooth Periodic Signal

Use the real Fourier basis and consider $f_s(x)=\sum_{k=1}^{\infty}k^{-3}\sqrt{2}\cos(2\pi kx)$. Its coefficient at frequency $k$ is $\theta_k=k^{-3}$; all sine and constant coefficients are zero. Estimate each retained coefficient with independent mean-zero noise of variance $v=\sigma^2/n$, and retain frequencies $1,\ldots,K$.

The finite-expansion risk identity gives

E\|\hat f_K-f_s\|_2^2 = \sum_{k=K+1}^{\infty} k^{-6}+Kv.

The first term is the squared truncation bias; the second is the variance of the $K$ estimated coefficients. Since $x^{-6}$ is decreasing, $\int_{K+1}^{\infty}x^{-6}\,dx\leq\sum_{k>K}k^{-6}\leq\int_K^{\infty}x^{-6}\,dx$. Consequently,

\frac{1}{5(K+1)^5}+Kv \leq E\|\hat f_K-f_s\|_2^2 \leq \frac{1}{5K^5}+Kv.

For example, at $K=2$ the exact bias is $3^{-6}+4^{-6}+5^{-6}+\cdots$; it lies between $1/1215$ and $1/160$. The total risk adds $2v$. Increasing $K$ reduces the bias at an order of $K^{-5}$, but increases the variance linearly in $K$. This example illustrates how rapid coefficient decay makes global truncation effective for a smooth periodic signal.

Worked Example: Fourier Truncation for a Signal With a Jump

Let $f_J(x)=\mathbf{1}\{x\geq 1/3\}$ on $[0,1]$, represented by its periodic Fourier series. For nonzero integer $k$, its complex Fourier coefficient is $c_k=\int_{1/3}^{1}e^{-2\pi ikx}\,dx$. Evaluating the integral gives

c_k=\frac{e^{-2\pi ik/3}-1}{2\pi ik}, \qquad |c_k|^2=\frac{\sin^2(\pi k/3)}{\pi^2 k^2}.

The second equality follows from $|e^{-2\pi ik/3}-1|^2=4\sin^2(\pi k/3)$. The constant coefficient is $c_0=2/3$. A truncation keeping frequencies $|k|\leq K$ has squared bias

\sum_{|k|>K}|c_k|^2 = \frac{2}{\pi^2}\sum_{k>K}\frac{\sin^2(\pi k/3)}{k^2} \leq \frac{2}{\pi^2}\sum_{k>K}\frac{1}{k^2} \leq \frac{2}{\pi^2K}.

Here $\sin(\pi k/3)=0$ when $k$ is divisible by 3, and its squared value is $3/4$ for the other integer residues. In particular, the nonzero coefficients have magnitude proportional to $1/k$, not $1/k^3$ as in the smooth example. If the retained real Fourier coefficients each have variance $v$, their total variance is $v$ times the number of retained real coefficients, and the risk is that variance plus the displayed squared bias. The slower bias decay is the cost of representing a jump with global oscillatory functions.

Haar Coefficients: Smoothness and Localization

For a concrete localized basis, use the Haar wavelet on $[0,1]$. At level $j$ and location $k$, its support is the dyadic interval of length $h=2^{-j}$. The normalized wavelet is $h^{-1/2}$ on the left half of that interval, $-h^{-1/2}$ on the right half, and zero elsewhere. It has integral zero, so a coefficient depends on local changes in the signal rather than its local constant level.

For $f(x)=x$, let the interval start at $a$. Direct integration gives the Haar coefficient

\beta_{jk} = h^{-1/2}\left(\int_a^{a+h/2}x\,dx-\int_{a+h/2}^{a+h}x\,dx\right) =-\frac{h^{3/2}}{4} =-\frac{2^{-3j/2}}{4}.

There are $2^j$ coefficients at level $j$, so their total squared energy is $2^j(2^{-3j/2}/4)^2=2^{-2j}/16$. By contrast, for a step at a point $\tau$, only the dyadic interval containing $\tau$ can have a nonzero coefficient at a given level, provided $\tau$ is not exactly a dyadic boundary. If $t\in(0,1)$ is the jump's fractional position within that interval, integration on the two halves gives

|\beta_{jk}|=\min(t,1-t)\,2^{-j/2}\leq \frac{2^{-j/2}}{2}.

Thus a smooth linear signal has many small coefficients at each level, while a single jump affects at most one location per level and can produce a coefficient of order $2^{-j/2}$. This is spatial localization: the jump's wavelet coefficients are concentrated near its location rather than spread over all basis functions.

Hard-Threshold Risk for a Gaussian Coefficient

For a coefficient $\beta$, suppose $Y=\beta+\varepsilon$, where $\varepsilon\sim N(0,s^2)$. The hard-threshold estimator at threshold $\lambda>0$ is $\delta_\lambda(Y)=Y\mathbf{1}\{|Y|>\lambda\}$. If $|Y|\leq\lambda$, it sets the coefficient to zero; otherwise it retains the noisy value.

Result: Write $a=(-\lambda-\beta)/s$, $b=(\lambda-\beta)/s$, and let $\Phi$ and $\phi$ be the standard normal distribution function and density. Then $$ E[(\delta_\lambda(Y)-\beta)^2] = \beta^2\{\Phi(b)-\Phi(a)\} +s^2\{1-\Phi(b)+b\phi(b)+\Phi(a)-a\phi(a)\}. $$

On the event $-\lambda\leq Y\leq\lambda$, the estimator is zero and the squared error is $\beta^2$. Since $Y=\beta+sZ$, this event is $a\leq Z\leq b$, with probability $\Phi(b)-\Phi(a)$. On the retained event the estimation error is $\varepsilon=sZ$. The two retained regions are $Z>b$ and $Z<a$. Integration by parts, using $\phi'(z)=-z\phi(z)$, gives

\int_b^\infty z^2\phi(z)\,dz=1-\Phi(b)+b\phi(b), \qquad \int_{-\infty}^a z^2\phi(z)\,dz=\Phi(a)-a\phi(a).

Multiplying their sum by $s^2$ and adding the drop-event error proves the formula. This exact expression includes both the chance of discarding a true coefficient and the noise in coefficients that survive thresholding.

Worked Example: Thresholded Haar Risks for a Smooth Signal and a Jump

Use a Haar expansion with the constant scaling function and the level-zero wavelet only. Estimate the scaling coefficient without thresholding, and hard-threshold the one wavelet coefficient. Let both coefficient estimates have independent Gaussian errors with $s=0.1$, and take $\lambda=0.2$. The omitted finer-scale coefficients contribute their exact squared approximation error.

Smooth signal $f(x)=x$. The scaling coefficient is $1/2$, the level-zero Haar coefficient is $-1/4$, and the omitted squared energy is $1/48$: the signal's centered energy is $\int_0^1(x-1/2)^2dx=1/12$, of which the retained wavelet accounts for $1/16$, leaving $1/12-1/16=1/48$. The scaling estimate contributes variance $s^2=0.01$.

The threshold risk is unchanged if the coefficient's sign is reversed, so use $\beta=0.25$. Then $a=-4.5$, $b=-0.5$, and the drop probability is $\Phi(-0.5)-\Phi(-4.5)\approx0.308534$. The retained-noise contribution is $0.01[1-\Phi(-0.5)-0.5\phi(0.5)+\Phi(-4.5)+4.5\phi(4.5)]\approx0.005155$. The drop contribution is $0.25^2(0.308534)\approx0.019283$, so the wavelet coefficient risk is approximately $0.024438$. Total risk is therefore $1/48+0.01+0.024438\approx0.05527$.

Jump signal $f(x)=\mathbf{1}\{x\geq1/3\}$. Its scaling coefficient is $2/3$, its level-zero Haar coefficient is $-1/3$, and its omitted squared energy is $1/9$. To verify the last value, the centered signal has energy $(2/3)(1/3)=2/9$, while the retained wavelet coefficient has squared magnitude $1/9$; the remainder is $2/9-1/9=1/9$. The scaling estimate again contributes $0.01$.

Set $\beta=1/3$ in the threshold formula. Now $a\approx-5.3333$, $b\approx-1.3333$, and the drop probability is approximately $0.09121$. The retained-noise contribution is approximately $0.01[1-\Phi(-1.3333)-1.3333\phi(1.3333)+\Phi(-5.3333)+5.3333\phi(5.3333)]\approx0.00690$. The drop contribution is $(1/9)(0.09121)\approx0.01013$, giving wavelet coefficient risk about $0.01704$. Total risk is $1/9+0.01+0.01704\approx0.13815$.

At this coarse resolution the jump's larger omitted energy dominates its total risk, even though thresholding makes the retained coefficient's risk relatively small. Localization does not eliminate approximation error; it changes how the signal's structure is allocated across coefficients and locations.

How to Read the Comparison

The Fourier calculations show the role of coefficient decay directly: the smooth periodic example has a summable tail of order $K^{-5}$, whereas the jump's global Fourier tail is only bounded at order $K^{-1}$. In the Haar representation, the smooth linear signal has energy at each level spread over $2^j$ coefficients, while a jump contributes at most one nonzero coefficient per level. Thresholding can therefore retain local evidence of a jump without retaining every coefficient at that resolution.

These examples do not imply that one basis always has lower risk. Risk depends on the signal, the resolution, the noise level, and the threshold. A Fourier truncation with too many frequencies pays a large variance cost; a wavelet threshold that is too high can discard genuine coefficients. The useful comparison is between the signal's coefficient structure and the estimator's bias-variance tradeoff.

Common Mistake: Applying Parseval to an Infinite Noisy Series

A tempting argument is to retain every coefficient, write the risk as the sum of coefficient variances, and call the result an $L^2$ risk. That is not automatically valid. If infinitely many retained coefficients each have independent Gaussian error with variance $v>0$, then the expected sum of squared errors is $\sum_m v=\infty$. More strongly, the error sequence is almost surely not square-summable: for independent errors, the events $\{|\varepsilon_m|>\sqrt{v}/2\}$ have the same positive probability, so infinitely many occur almost surely. The resulting coefficient sequence does not define an $L^2$-valued estimator.

The finite-set risk identity avoids this problem because the estimated part is a finite sum and the omitted part is the $L^2$ tail of the true signal. Infinite-series risk claims require an explicit convention and conditions ensuring the estimator exists; they cannot be obtained by applying the finite-set formula without qualification.

Key takeaway: Fourier truncation pays for omitted high-frequency energy and for every retained noisy coefficient. Wavelet thresholding makes coefficient decisions locally, which can suit signals with jumps. In either basis, state the retained set or resolution before calculating risk.

Check Your Understanding

Use the calculations and definitions above to answer these questions.

For the smooth Fourier signal with coefficients $k^{-3}$, what are the squared-bias and variance terms when frequencies $1,\ldots,K$ are retained?
Why does the Fourier coefficient magnitude of the step signal decay more slowly than that of the smooth periodic example?
At Haar level $j$, how many coefficients can be affected by one jump, and what is an upper bound for the magnitude of such a coefficient?
In the hard-threshold risk formula, which term accounts for the possibility of discarding a true nonzero coefficient?
Why does the finite orthonormal risk identity not by itself justify an $L^2$ estimator formed from infinitely many noisy coefficients of common positive variance?