Diagnosing Failures of Adaptive Confidence Bands

Why a Reported Confidence Level Can Mislead

A confidence band is meant to cover a function, not just one of its values. Yet a band may be assembled from intervals calibrated separately at each location, centered at a biased smoother, or justified only under a smoothness condition that the regression function does not satisfy. In each case, the reported confidence level can conceal a substantially weaker guarantee.

The useful diagnostic is to separate three questions: Is the calibration simultaneous over the domain? Is the estimator's bias small relative to its standard error? Does the regression function actually belong to the smoothness class used to justify the band? These are distinct issues; correcting one does not automatically correct the others.

Definition: Let \(f\) be a regression function on a domain \(\mathcal X\), and let a random band be \(C_n(x)=[L_n(x),U_n(x)]\). Its pointwise coverage at \(x\) is \(P_f\{f(x)\in C_n(x)\}\). Its simultaneous coverage is \(P_f\{f(x)\in C_n(x)\text{ for every }x\in\mathcal X\}\). A band is honest over a function class \(\mathcal F\) at level \(1-\alpha\) if \[ \inf_{f\in\mathcal F}P_f\{f(x)\in C_n(x)\text{ for every }x\in\mathcal X\}\geq 1-\alpha. \]

Pointwise coverage concerns one fixed location at a time. Simultaneous coverage concerns a single event that must hold across the entire domain. The latter is the relevant guarantee when the band is presented as covering the whole regression curve.

Pointwise Calibration Does Not Guarantee a Band

The distinction already appears on a domain with only two points. Suppose the estimation errors at the two points are independent standard normal random variables, and at each point we report the estimate plus or minus 1.96 standard errors. Each interval has approximately 95% coverage. But the probability that both cover is approximately \(0.95^2\), not 0.95.

Worked Example: Two Pointwise Intervals, One Undercovering Band

Let \(\mathcal X=\{x_1,x_2\}\), and suppose the two estimates satisfy \(\hat f(x_j)-f(x_j)=Z_j\), where \(Z_1,Z_2\) are independent \(N(0,1)\) variables. At each point use \(C_n(x_j)=[\hat f(x_j)-1.96,\hat f(x_j)+1.96]\).

For either point separately, the coverage probability is \[ P(|Z_j|\leq 1.96)=2\Phi(1.96)-1\approx 0.9500. \] For the band to cover both points, both errors must be within 1.96. Independence gives \[ P(|Z_1|\leq1.96,\ |Z_2|\leq1.96) =\{2\Phi(1.96)-1\}^2 \approx 0.9500^2 \approx 0.9025. \] Thus two intervals with approximately 95% pointwise coverage produce only about 90.25% simultaneous coverage. The shortfall is caused by calibrating each point separately, not by bias.

Result: Suppose \(m\) locations each have a centered, standard normal estimation error, and each interval uses the common radius \(c\) standard errors. By the union bound, \[ P\{\text{at least one location is missed}\} \leq \sum_{j=1}^m P(|Z_j|>c). \] In particular, choosing \(c=z_{1-\alpha/(2m)}\), where \(\Phi(z_q)=q\), makes the probability of any miss at most \(\alpha\). This bound does not require independence.

To verify the result, the two tails at location \(j\) have total probability \(2\{1-\Phi(c)\}=\alpha/m\) for this choice of \(c\). The probability of a union of events is at most the sum of their probabilities, so the probability of at least one miss is at most \(m(\alpha/m)=\alpha\). Taking complements gives simultaneous coverage of at least \(1-\alpha\). For a continuous domain, replacing a finite maximum by a supremum requires its own calibration argument; a finite-grid calculation alone does not establish coverage between grid points.

Ignored Smoothing Bias

A pointwise interval can also fail even when there is just one target location and the standard error is computed correctly. The usual symmetric Gaussian interval around a smoother is calibrated as if the smoother's expectation equaled the target. When its bias is substantial relative to its standard error, the interval is shifted away from the target.

Result: Suppose \(\hat\theta\sim N(\theta+b,\tau^2)\), where \(b\) is its bias and \(\tau>0\) is its standard deviation. The interval \([\hat\theta-c\tau,\hat\theta+c\tau]\) covers \(\theta\) with probability \[ \Phi\!\left(c-\frac{b}{\tau}\right) -\Phi\!\left(-c-\frac{b}{\tau}\right). \] In particular, the nominal centered-normal coverage \(2\Phi(c)-1\) applies when \(b=0\); more generally it is approached if \(b/\tau\to0\).

For the derivation, write \(\hat\theta=\theta+b+\tau Z\), with \(Z\sim N(0,1)\). Coverage is equivalent to \(|\hat\theta-\theta|\leq c\tau\), or \[ -c-\frac{b}{\tau}\leq Z\leq c-\frac{b}{\tau}. \] Taking the difference of the standard normal probabilities at these two endpoints gives the stated expression. If \(b/\tau\to0\), continuity of \(\Phi\) gives the centered-normal limit. The calculation shows why a small bias in absolute units is not enough: what matters for this interval is the bias relative to \(\tau\).

Worked Example: A Smoother Whose Bias Dominates Its Standard Error

Use the Gaussian regression model \(Y_i=f(x_i)+\varepsilon_i\) at the three design points \(x_0=0,x_1=1,x_2=2\), with independent errors \(\varepsilon_i\sim N(0,1)\). Let \(f(0)=0,\ f(1)=1,\ f(2)=3\), and estimate \(f(0)\) by \(\hat f(0)=(Y_0+Y_1+Y_2)/3\).

The expectation is \(\{0+1+3\}/3=4/3\), so the bias for \(f(0)\) is \(b=4/3\). Independence and unit error variances give \(\operatorname{Var}\{\hat f(0)\}=(1+1+1)/9=1/3\), hence \(\tau=1/\sqrt3\). The nominal 95% interval uses radius \(r=1.96/\sqrt3\).

Its actual coverage is \[ P_f(-r\leq\hat f(0)\leq r) =\Phi\!\left(\frac{r-4/3}{1/\sqrt3}\right) -\Phi\!\left(\frac{-r-4/3}{1/\sqrt3}\right). \] The standardized endpoints are \(1.96-4\sqrt3/3\approx-0.3494011\) and \(-1.96-4\sqrt3/3\approx-4.2694011\). Therefore the coverage is \(\Phi(-0.3494011)-\Phi(-4.2694011)\approx 0.3634\), not 0.95. The interval is centered at an estimator with expectation \(4/3\), while the target is zero; its standard error does not account for that displacement.

This example is deliberately simple, but the diagnostic applies to smoothing methods generally. A variance estimate describes random fluctuation around the estimator's expectation; it does not, by itself, describe the distance from that expectation to the target function. A band that ignores smoothing bias can be narrow and systematically displaced.

Unsupported Smoothness Is a Coverage Failure

Adaptive bands often depend on assumptions that constrain how rapidly \(f\) can vary. Such assumptions are not merely technical conveniences: they restrict which functions the guarantee covers. If the true function is outside the assumed class, a coverage statement proved over that class does not automatically extend to it.

Worked Example: A Narrow Band Cannot Be Justified by Unobserved Smoothness

Suppose observations are taken only at \(x=0\) and \(x=1\), with independent errors having the same distribution under every regression function. Consider \(f_0(x)=0\) and the triangular function \[ f_A(x)=2A\min(x,1-x),\qquad 0\leq x\leq1, \] where \(A>0\). Both functions equal zero at the observed design points, but \(f_A(1/2)=A\), whereas \(f_0(1/2)=0\). Their data distributions are therefore identical.

Suppose a band claims coverage at least \(1-\alpha\) for both functions. Under their common data distribution, let \(E_0\) be the event that the band covers \(f_0\) everywhere and \(E_A\) the event that it covers \(f_A\) everywhere. Each event has probability at least \(1-\alpha\). Since \[ P(E_0\cap E_A)\geq P(E_0)+P(E_A)-1\geq1-2\alpha, \] with probability at least \(1-2\alpha\) the band at \(x=1/2\) must contain both 0 and \(A\). Its width there must then be at least \(A\).

The function \(f_A\) has Lipschitz constant \(2A\), so choosing \(A\) large can violate a proposed small Lipschitz bound while leaving the observations' distribution unchanged. Thus the data at these design points cannot distinguish the functions or support a uniformly narrow band over a class that includes both. A narrow-band guarantee requires the smoothness restriction to be part of the actual scope of the guarantee, not an unverified description of the truth.

A Practical Diagnostic

When a band appears to undercover, identify which event the reported probability describes and which assumptions make that probability valid. The following sequence separates the main failure modes.

Specify the coverage event.
Check whether the claim concerns one fixed \(x\) or all \(x\in\mathcal X\). Pointwise intervals do not by themselves provide simultaneous calibration.

Compare bias with uncertainty.
For a smoother, assess whether its bias is negligible relative to its standard error. If not, a variance-only radius can miss the target even at one location.

Check the function-class claim.
State the smoothness restrictions required by the guarantee and ask whether they are justified. A result over a restricted class is not a guarantee for functions outside it.

A frequent mistake is to reason that a band is valid because its pointwise intervals each have the advertised coverage. That reasoning neglects the probability of at least one miss across locations; the two-point example gives a direct counterexample. Another mistake is to treat a small standard error as evidence of small total error. The bias calculation shows that uncertainty around the estimator's expectation and error relative to the target are different quantities. Finally, a data-driven choice of smoothness cannot make an unsupported smoothness assumption true.

Key takeaway: Diagnose undercoverage by checking three separate ingredients: simultaneous rather than merely pointwise calibration, bias control on the scale of the standard error, and a smoothness class that genuinely covers the regression functions for which the guarantee is claimed.

Check Your Understanding

For each question, identify the coverage event or assumption at issue.

If each of four independent pointwise intervals has exactly 95% coverage, what is the probability that all four cover? Why is it not 95%?
For a Gaussian estimator with bias \(b\) and standard error \(\tau\), which quantity determines whether bias is negligible for a symmetric interval: \(|b|\) or \(|b|/\tau\)? Explain.
In the smoothing example, verify the estimator's expectation and variance from the three response means and error variances.
Why can two regression functions that agree at all observed design points still prevent a uniformly narrow band between those points?
Does a coverage theorem over a Lipschitz class automatically apply to a function whose Lipschitz constant exceeds the class bound? State why or why not.