Lepski’s Method for Selecting a Smoothing Scale

Why Compare Estimates Across Bandwidths?

At a fixed point \(x_0\), a kernel estimate depends on its bandwidth. A narrow bandwidth uses observations close to \(x_0\), so it can track local features but is relatively noisy. A wider bandwidth averages more observations and is typically less noisy, but can smooth away local variation. Lepski’s method uses the estimates themselves to decide how far the bandwidth can be increased without creating a discrepancy that is too large to explain by stochastic error.

The method requires a finite, ordered collection of bandwidths and thresholds matched to the noise scale of each estimate. Its selection rule is precise: choose the largest bandwidth whose estimate agrees, within the specified thresholds, with every estimate at a finer bandwidth. This tutorial develops that construction and its basic stochastic justification; it does not require knowing the unknown smoothness of the regression function.

Pointwise Kernel Estimates and Their Error Scales

For concreteness, consider fixed design points \(x_1,\ldots,x_n\) and observations \(Y_i=f(x_i)+\varepsilon_i\). The errors are independent, have mean zero, and are sub-Gaussian with scale \(\sigma\): for every real \(t\), \(E[\exp(t\varepsilon_i)]\leq \exp(\sigma^2t^2/2)\). The design points are treated as fixed. Let \(K\) be a nonnegative kernel supported on \([-1,1]\), and suppose its denominator below is positive for each bandwidth under consideration.

For a regular design and a bounded kernel, a usual weight condition is \(\sum_i w_{i,h}(x_0)^2\leq C/(nh)\), where \(C\) is a constant independent of \(h\) and \(n\). The sub-Gaussian error assumption then gives a stochastic scale no larger than \(\sigma\sqrt{C/(nh)}\). More generally, the construction only needs known upper bounds \(v_j\) on the scales of the centered estimates, with those upper bounds nonincreasing as the bandwidth increases.

Choose bandwidths \(h_1< h_2<\cdots<h_M\), from finest to coarsest. Let \(v_j\) be a positive, nonincreasing upper bound on the stochastic scale of \(\hat f_{h_j}(x_0)\). For example, under the weight condition one may take \(v_j=\sigma\sqrt{C/(nh_j)}\). A high-probability multiplier \(q\) gives the comparison threshold \(\tau_j=2qv_j\) for the \(j\)th, finer estimate in a comparison.

The threshold is indexed by \(k\), the finer bandwidth in the pair. Its estimate generally has the larger stochastic scale. Thus a candidate coarse estimate is checked against each finer estimate using that finer estimate’s threshold. A comparison against only the immediately preceding bandwidth is not the rule: every finer estimate must pass.

Worked Comparisons

Why the Threshold Controls Stochastic Disagreement

The basic calibration argument needs no independence between the kernel estimates: they are computed from the same observations and are generally dependent. A simultaneous bound for each centered estimate is enough. The next result states the guarantee explicitly.

Proof. For each \(j\), apply the assumed tail bound with \(u=q\): \(P(|\xi_{h_j}(x_0)|>qv_j)\leq2\exp(-q^2/2)=\delta/M\). The union bound therefore gives \(P(\text{some }j\text{ has }|\xi_{h_j}(x_0)|>qv_j)\leq \sum_{j=1}^M\delta/M=\delta\). On the complementary event, take any \(k<j\). Since \(v_j\leq v_k\), the triangle inequality yields \[ |\xi_{h_j}(x_0)-\xi_{h_k}(x_0)| \leq|\xi_{h_j}(x_0)|+|\xi_{h_k}(x_0)| \leq qv_j+qv_k \leq2qv_k=\tau_k. \] This holds for every pair on the same event, whose probability is at least \(1-\delta\). \(\square\)

For the fixed-design model above, the tail condition follows from the sub-Gaussian assumption. Indeed, independence gives \(E[\exp(t\xi_h)]=\prod_i E[\exp(tw_{i,h}\varepsilon_i)] \leq\exp(t^2\sigma^2\sum_iw_{i,h}^2/2)\). Thus a valid scale is \(v_h=\sigma\sqrt{\sum_iw_{i,h}^2}\), or any upper bound on it. If the chosen upper bounds decrease with bandwidth, the theorem applies. The union bound accounts for all \(M\) estimates and does not require their errors to be independent.

What the Comparison Means—and What It Does Not Mean

Write \(m_j=m_{h_j}(x_0)\) and \(\xi_j=\xi_{h_j}(x_0)\), so that \(\hat f_{h_j}(x_0)=m_j+\xi_j\). For a pair \(k<j\), the observed difference is the sum of a mean difference and a stochastic difference. On the simultaneous event in the theorem, the stochastic part is at most \(\tau_k\). Thus a discrepancy beyond \(\tau_k\) cannot be explained by that event’s stochastic fluctuation alone; it signals a difference between the estimates’ means as well.

A useful deterministic implication makes the converse direction precise. If for every \(k<j\), \(|m_j-m_k|\leq\tau_k-2qv_k\), and the right-hand sides are nonnegative, then index \(j\) passes on the simultaneous event. In fact, on that event, \[ |\hat f_{h_j}(x_0)-\hat f_{h_k}(x_0)| \leq|m_j-m_k|+|\xi_j-\xi_k| \leq(\tau_k-2qv_k)+2qv_k =\tau_k. \] This is a comparison fact, not an oracle bound: it does not by itself quantify the risk of the selected estimate or establish adaptation over a function class.

If all candidate estimates have the same mean \(m\), then every mean difference is zero. On the simultaneous event, every pairwise comparison passes, so the selector chooses \(h_M\), the coarsest candidate. This follows directly from the comparison guarantee and the maximum-index rule. If means separate as bandwidth increases, some coarse candidates may fail, and the rule returns the coarsest one that still agrees with all finer estimates at the prescribed stochastic scale.

Common Mistakes and Practical Checks

• Comparing only adjacent bandwidths. Passing the comparison with \(h_{j-1}\) does not imply passing comparisons with every \(h_k\), \(k<j-1\). The definition requires all finer estimates.
• Using the candidate’s coarse scale for every threshold. The threshold for comparing with \(h_k\) is based on the finer estimate’s scale \(v_k\), not simply on the candidate bandwidth’s scale.
• Reversing the order. Here \(h_1\) is the finest bandwidth and \(h_M\) the coarsest. The rule maximizes the index, not the amount of local detail.
• Treating the threshold as a bias bound. It calibrates stochastic differences. The rule does not know the mean differences or guarantee that every passing estimate has small bias.
• Miscomputing a pairwise difference. Recalculate each absolute difference from the displayed estimates before comparing it with its threshold. For example, if the estimates are \(1.90,1.90,1.80\), the last estimate differs from each of the first two by \(0.10\), not by \(0.00\).

Lepski’s selector is useful because its comparisons use observable estimates and thresholds rather than the unknown smoothness level. The stochastic calibration explains why the comparisons are meaningful; evaluating the risk and adaptation properties of the selected bandwidth requires additional analysis.