Worked Problems in Adaptive Estimation and Honest Uncertainty

Four Calculations That Clarify Adaptation

Adaptive procedures use the data to choose how much to smooth, which coefficients to retain, or how wide an uncertainty band should be. These choices can improve performance when the underlying function is unknown. They do not, by themselves, guarantee that the procedure is close to an oracle choice or that a reported band has its nominal coverage. The calculations below separate these claims: a comparison rule can control estimation error under explicit bounds; threshold risk can be evaluated coefficient by coefficient; honesty must be checked over the full stated class; and honest adaptation needs conditions that make the unknown bias sufficiently identifiable.

The examples use simplified settings to keep each calculation visible. A result proved for a scalar or sequence problem should not be mistaken for a general guarantee for every regression model.

Worked Problem 1: Lepski Comparisons and an Oracle Bound

Consider a finite, ordered set of bandwidths \(h_1<\cdots<h_M\). Write the corresponding estimators at a fixed location as \(\hat f_j=f+b_j+\xi_j\), where \(b_j\) is the bias and \(\xi_j\) is the random error. Smaller bandwidths are less biased but noisier. Suppose that, on an event of interest, \(|b_j|\le B_j\) and \(|\xi_j|\le\lambda s_j\), where \(B_j\) is nondecreasing and \(s_j\) is nonincreasing in \(j\). Here \(s_j\) is a noise scale, and \(\lambda\) is a fixed bound on the noise in those units.

Definition: For a comparison threshold \(\kappa>0\), say index \(j\) passes the Lepski comparisons if \(|\hat f_j-\hat f_l|\le\kappa s_l\) for every \(l<j\). Select the largest passing index, denoted \(\hat j\). The rule prefers the largest bandwidth whose estimate remains sufficiently close to estimates at all finer bandwidths.

The following deterministic result explains when that rule is useful. Its assumptions must hold on the same event for every candidate bandwidth.

Theorem: Suppose \(j_0\) is the largest index satisfying \(B_{j_0}\le s_{j_0}\), and suppose \(|b_j|\le B_j\), \(|\xi_j|\le\lambda s_j\), with \(B_j\) nondecreasing and \(s_j\) nonincreasing. If \(\kappa\ge 2+2\lambda\), then \(j_0\) passes the comparisons, \(\hat j\ge j_0\), and \[ |\hat f_{\hat j}-f|\le(\kappa+1+\lambda)s_{j_0}. \]

Proof. For any \(l<j_0\), monotonicity and the definition of \(j_0\) give \(B_l\le B_{j_0}\le s_{j_0}\le s_l\). Therefore \[ |\hat f_{j_0}-\hat f_l| \le |b_{j_0}|+|b_l|+|\xi_{j_0}|+|\xi_l| \le 2s_l+2\lambda s_l \le\kappa s_l. \] Thus \(j_0\) passes and the largest passing index satisfies \(\hat j\ge j_0\). If \(\hat j>j_0\), its comparison with \(j_0\) gives \(|\hat f_{\hat j}-\hat f_{j_0}|\le\kappa s_{j_0}\); if \(\hat j=j_0\), that difference is zero. Also, \[ |\hat f_{j_0}-f|\le B_{j_0}+\lambda s_{j_0} \le(1+\lambda)s_{j_0}. \] The triangle inequality proves the stated bound. \(\square\)

Worked Example: Selecting Among Three Bandwidths

Take \(n=100\), bandwidths \(h=0.1,0.2,0.4\), bias bounds \(B(h)=h\), and noise scales \(s(h)=1/\sqrt{nh}\). Suppose the observed estimates are \(1.02,1.00,1.05\), respectively. Use comparison threshold \(\kappa=2\).

The three noise scales are \(s(0.1)=1/\sqrt{10}\approx0.3162\), \(s(0.2)=1/\sqrt{20}\approx0.2236\), and \(s(0.4)=1/\sqrt{40}\approx0.1581\). The \(h=0.2\) estimate passes because its only comparison is \(|1.00-1.02|=0.02\le2(0.3162)=0.6324\).

The \(h=0.4\) estimate must be compared with both finer estimates. The differences are \(|1.05-1.00|=0.05\le2(0.2236)=0.4472\) and \(|1.05-1.02|=0.03\le2(0.3162)=0.6324\). It passes too, so the rule selects \(h=0.4\), the largest candidate.

For comparison, the largest bandwidth with \(B(h)\le s(h)\) is \(0.2\): \(0.2\le0.2236\), whereas \(0.4>0.1581\). The theorem explains how, under its simultaneous bias and noise bounds and a sufficiently large threshold, choosing beyond this balance point can still have error controlled on the scale \(s(0.2)\). The observed comparisons alone do not verify those bounds.

Worked Problem 2: Risk of a Hard-Thresholded Wavelet Coefficient

In a Gaussian sequence model for one wavelet coefficient, observe \(Y=\theta+\sigma Z\), where \(Z\sim N(0,1)\), and use hard thresholding \(\hat\theta=Y\mathbf{1}\{|Y|>t\}\). The coefficient is retained only when its observed magnitude exceeds \(t\). The exact risk makes clear that the threshold has different effects near zero and far from zero.

Result: Let \(a=(-t-\theta)/\sigma\), \(b=(t-\theta)/\sigma\), and let \(\phi\) and \(\Phi\) be the standard normal density and distribution function. For \(t\ge0\) and \(\sigma>0\), \[ E[(\hat\theta-\theta)^2] =\theta^2\{\Phi(b)-\Phi(a)\} +\sigma^2\{\Phi(a)-a\phi(a)+1-\Phi(b)+b\phi(b)\}. \]

Derivation. The coefficient is discarded precisely when \(-t\le Y\le t\), which is the event \(a\le Z\le b\). On that event, the squared error is \(\theta^2\), giving the first term. When it is retained, the error is \(Y-\theta=\sigma Z\). The retained event is \(Z<a\) or \(Z>b\). Integration by parts gives \(\int_{-\infty}^{a}z^2\phi(z)\,dz=\Phi(a)-a\phi(a)\) and \(\int_b^\infty z^2\phi(z)\,dz=1-\Phi(b)+b\phi(b)\). Adding the two contributions proves the formula.

Worked Example: Risk at a Zero Coefficient

Set \(\theta=0\) and \(t=2\sigma\). Then \(a=-2\), \(b=2\), and symmetry gives \[ E[\hat\theta^2] =2\sigma^2\{2\phi(2)+1-\Phi(2)\}. \] Using \(\phi(2)\approx0.053991\) and \(1-\Phi(2)\approx0.022750\), the risk is \(2\sigma^2(0.107982+0.022750)\approx0.26146\sigma^2\).

If \(\sigma=0.5\), this is approximately \(0.26146(0.25)=0.06537\). Thresholding does not make the risk zero: noise occasionally pushes \(Y\) past the threshold, causing the coefficient to be retained. This retained-noise contribution is exactly what the tail terms in the risk calculation measure.

Worked Problem 3: Checking Honesty Across Nested Classes

Let \(\mathcal F_{\mathrm{s}}\subseteq\mathcal F_{\mathrm{r}}\) be a smoother class inside a rougher, larger class. A band \(C_n\) is honest over a class \(\mathcal F\) at level \(1-\alpha\) if \[ \inf_{f\in\mathcal F}P_f\{f(x)\in C_n(x)\text{ for every }x\}\ge1-\alpha. \] This is a uniform statement: the coverage probability must meet the target for every function in the class, not merely on average or at a selected function. Honesty over \(\mathcal F_{\mathrm{r}}\) implies honesty over its subset \(\mathcal F_{\mathrm{s}}\), but the reverse implication does not hold.

Worked Example: A Band That Covers the Smooth Case but Fails on the Larger Class

Consider a scalar parameter \(f\), which can also be viewed as a constant function. Let \(\mathcal F_{\mathrm{s}}=\{0\}\) and \(\mathcal F_{\mathrm{r}}=\{0,2\}\). Suppose the reported interval is \(C(X)=[X-1.96,X+1.96]\). At \(f=0\), let \(X=Z\) with \(Z\sim N(0,1)\). Then \(P_0\{0\in C(X)\}=P(|Z|\le1.96)\approx0.95\).

At \(f=2\), suppose the estimator has bias \(2\), so \(X=4+Z\). The interval covers \(f=2\) exactly when \(|X-2|\le1.96\), or \(|Z+2|\le1.96\). Thus \[ P_2\{2\in C(X)\} =P(-3.96\le Z\le-0.04) =\Phi(-0.04)-\Phi(-3.96) \approx0.4840. \]

The interval has about 95% coverage at the sole smooth-class parameter, but only about 48% at the second parameter in the rougher class. It is not an honest 95% interval over \(\mathcal F_{\mathrm{r}}\). Checking a smooth submodel, or reporting coverage at a typical function, cannot establish uniform honesty.

When Can Adaptation Be Honest?

The difficulty is not simply that the smoothness label is unknown. A band must account for bias, and a data-driven choice of bandwidth can make that bias hard to distinguish from noise. If a rough function is statistically difficult to tell from a smooth one, a band narrow enough for the smooth case may miss the rough function too often. Unrestricted adaptation over nested smoothness classes can therefore be impossible at the desired coverage and width simultaneously; the earlier material on honest adaptive bands develops that obstruction.

Adaptation becomes more plausible when the allowed function class supplies information about bias that the data can reveal. Examples include self-similarity restrictions, which require signal across relevant scales, and polished-tail restrictions in sequence models, which prevent the unresolved tail from behaving arbitrarily. A known lower bound on smoothness or a sufficiently separated collection of classes can also help. These are restrictions on the model class, not automatic consequences of choosing a data-dependent bandwidth.

Result: Suppose a procedure selects an estimator \(\hat f_{\hat h}\) and a radius \(R_n(x)\). If, uniformly over a restricted class \(\mathcal F_*\), an event \(E_f\) has probability at least \(1-\alpha\), and on that event \[ |\hat f_{\hat h}(x)-f(x)|\le R_n(x)\quad\text{for every }x, \] then the band \([\hat f_{\hat h}(x)-R_n(x),\hat f_{\hat h}(x)+R_n(x)]\) is honest over \(\mathcal F_*\) at level \(1-\alpha\).

Proof. On \(E_f\), the displayed error bound implies \(\hat f_{\hat h}(x)-R_n(x)\le f(x)\le\hat f_{\hat h}(x)+R_n(x)\) for every \(x\). Thus \(E_f\) is contained in the event that the entire function lies in the band. Its probability is at least \(1-\alpha\), uniformly for \(f\in\mathcal F_*\); taking the infimum over that class proves the claim. \(\square\)

The implication is simple; obtaining its premise is the hard part. A simultaneous stochastic bound controls noise across locations and candidate bandwidths. A bias bound must also hold at the selected bandwidth. A self-similarity or related restriction can make scale comparisons informative about that bias, while a polished-tail condition can constrain how much unobserved fine-scale energy remains. If the bias bound or the probability event fails for even part of the claimed class, the coverage conclusion does not follow.

Common Pitfalls

Treating the selected bandwidth as an oracle. Passing comparisons does not prove that the selected estimate has the smallest risk. The oracle bound above also needs simultaneous bias and noise bounds.
Ignoring thresholding’s retained-noise risk. A zero coefficient can still produce a nonzero estimate when noise crosses the threshold.
Checking coverage only on the smoother subclass. Uniform honesty over a larger class requires coverage control at every member of that larger class.
Calling any smoothness restriction sufficient for adaptation. A restriction helps only if it supports a valid bias bound or makes relevant scales distinguishable; the coverage guarantee must still be proved.

Key takeaway: Adaptive estimation, threshold risk, and honest uncertainty are related but distinct calculations. Lepski comparisons need verifiable bias and noise control; coefficient risk includes both discarded-signal and retained-noise terms; and a band’s honesty is determined by its worst coverage over the entire stated class.

Check Your Understanding

Use the definitions and calculations above to answer the following.

In the Lepski theorem, why must the comparison threshold depend on the noise bound \(\lambda\)?
For a zero wavelet coefficient, which event produces nonzero hard-thresholding error?
Why does honesty over a rough class imply honesty over a nested smooth subclass, but not conversely?
In the interval example, write the event for coverage at \(f=2\) in terms of \(Z\).
Name one kind of function-class restriction that can help make adaptive uncertainty quantification possible, and state what it must help control.