Gaussian Threshold Calibration for Wavelet Regression

Why Thresholds Depend on the Noise and the Number of Coefficients

A wavelet threshold is intended to separate coefficients carrying signal from fluctuations caused by noise. Even when every true coefficient is zero, a large collection of noisy estimates will usually contain some large values. A threshold that seems adequate for one coefficient can therefore be too permissive when applied to thousands. Calibration must account for both the noise scale and the number of coefficients being examined.

The goal here is a precise Gaussian benchmark: choose thresholds so that the probability of any pure-noise coefficient exceeding its threshold is controlled. This is simultaneous error control. It differs from calibrating the probability for one coefficient, and it does not require the coefficient noises to be independent.

For an orthonormal transform of independent Gaussian observation errors with common variance, the transformed errors are again Gaussian with the same variance and are independent. In applications, the coefficient-estimation variance may instead be \(\sigma^2/n\), or may vary by scale. The threshold must use the standard deviation of the coefficient error itself, not automatically the standard deviation of an individual observation.

A Gaussian Tail Bound

The basic calibration follows from a tail inequality for a standard normal random variable. The bound below is deliberately simple; its advantage is that it gives an explicit finite-sample guarantee after applying the union bound.

Proof. The moment-generating function of \(Z\) is \(E[\exp(sZ)]=\exp(s^2/2)\). For \(x>0\) and \(s>0\), Markov's inequality gives \(P(Z>x)=P(\exp(sZ)>\exp(sx))\leq \exp(-sx)E[\exp(sZ)]=\exp(-sx+s^2/2)\). Taking \(s=x\), which is positive because this part of the proof assumes \(x>0\), yields \(P(Z>x)\leq\exp(-x^2/2)\). The same argument applied to \(-Z\), which also has distribution \(N(0,1)\), gives \(P(Z<-x)\leq\exp(-x^2/2)\). Adding the two bounds proves the result for \(x>0\). At \(x=0\), \(P(|Z|>0)\leq 1\leq 2\), so the stated bound holds there as well. \(\square\)

Simultaneous Calibration and the Universal Threshold

When all \(M\) coefficients have error variance \(\sigma^2\), standardize each error as \(Z_i=\varepsilon_i/\sigma\). Each \(Z_i\) is standard normal. For any \(\lambda\geq0\), the union bound and the Gaussian tail inequality imply \(P(\max_i|\varepsilon_i|>\lambda)\leq 2M\exp[-\lambda^2/(2\sigma^2)]\). This argument does not assume independence: the union bound applies to any dependence structure among the errors.

Proof. Apply the union bound, then substitute the displayed threshold: \(P(\max_i|\varepsilon_i|>\lambda_\alpha)\leq\sum_{i=1}^M P(|\varepsilon_i|>\lambda_\alpha)\leq2M\exp[-\lambda_\alpha^2/(2\sigma^2)]\). Because \(\lambda_\alpha^2/(2\sigma^2)=\log(2M/\alpha)\), the right-hand side equals \(2M\exp[-\log(2M/\alpha)]=\alpha\). \(\square\)

The conventional universal threshold is \(\sigma\sqrt{2\log M}\) when \(M>1\). It describes the leading size of the largest among many Gaussian noise coefficients. It is not the same as the finite-sample threshold above for an arbitrary specified \(\alpha\): the latter includes the factor \(2/\alpha\) inside the logarithm to provide the stated union-bound guarantee. The universal threshold is a useful reference scale, not a promise of a particular finite-sample exceedance probability.

The leading scale of the universal threshold can also be justified asymptotically when the errors are independent. Let \(Z_1,\ldots,Z_M\) be independent \(N(0,1)\) variables. Then \(\max_i|Z_i|/\sqrt{2\log M}\xrightarrow{p}1\) as \(M\to\infty\). For the upper bound, fix \(\epsilon>0\). The Gaussian tail inequality gives \(P(\max_i|Z_i|>(1+\epsilon)\sqrt{2\log M})\leq2M^{1-(1+\epsilon)^2}\), which tends to zero.

For the lower bound, put \(t=(1-\epsilon)\sqrt{2\log M}\), considering \(0<\epsilon<1\). For \(t>0\), the standard normal density \(\phi\) is decreasing on the positive half-line, so \(P(|Z_1|>t)\geq P(t\leq Z_1\leq t+1/t)\geq \phi(t+1/t)/t\). The last expression is \((2\pi)^{-1/2}t^{-1}\exp[-(t+1/t)^2/2]\). Consequently \(M P(|Z_1|>t)\to\infty\): its dominant power of \(M\) is \(M^{1-(1-\epsilon)^2}\), while the remaining factors have only logarithmic or constant order. Independence now gives \(P(\max_i|Z_i|\leq t)=(1-P(|Z_1|>t))^M\leq\exp[-M P(|Z_1|>t)]\to0\). Together these bounds establish the convergence in probability and explain why \(\sqrt{2\log M}\) is the natural leading scale. \(\square\)

Scale-Dependent Thresholds

Wavelet levels contain different numbers of coefficients, and their coefficient errors need not have the same variance. A single global threshold can therefore be unnecessarily severe at some levels or insufficiently calibrated at others. The same simultaneous argument can be applied separately at each level and the allowable error probability allocated across levels.

To see the final claim, apply the single-level bound at each level and then take a union bound over levels: \(P(\text{any exceedance})\leq\sum_jP(\text{an exceedance at level }j)\leq\sum_j\alpha_j\leq\alpha\). Thus the allocation determines how the total error budget is distributed. Equal allocation is simple, but other choices may be appropriate when levels have different roles. A larger \(m_j\) raises the threshold through the logarithm, and a smaller allocated \(\alpha_j\) raises it as well.

Interpretation, Limitations, and a Common Pitfall

Threshold calibration concerns the noise event \(\max_i|\hat\theta_i-\theta_i|\leq\lambda\). On this event, every coefficient estimation error is below its threshold. Thresholding rules use this comparison to decide which estimated coefficients to retain or shrink, but the probability bound alone does not establish that a particular threshold minimizes mean squared error or gives the best reconstruction. Those questions also depend on the true coefficient sizes and the loss being considered.

A frequent mistake is to use a one-coefficient Gaussian cutoff for every coefficient and then claim the same error probability for the entire wavelet expansion. If there are \(M\) coefficients, the probability of at least one exceedance can be much larger than the probability for one coefficient. The correction is to calibrate against the family size, either globally or level by level with an explicit error allocation.

A second practical issue is that \(\sigma\) or \(\sigma_j\) is often unknown. Replacing it with an estimate and inserting that estimate into the formula does not automatically preserve the stated Gaussian guarantee: the threshold is then random, and the preceding proof assumes the standard deviation used in the calculation is correct. Exact control with estimated noise requires an additional argument accounting for its uncertainty. Similarly, if Gaussianity is not justified, these Gaussian tail calculations should not be treated as distribution-free guarantees.

The central calibration principle is simple: identify the standard deviation of each coefficient error, count the coefficients covered by the claim, and specify the desired simultaneous error probability. The logarithm makes the threshold increase slowly with the number of coefficients, but ignoring that increase can still undermine a global noise-control claim.