Design Density Conditions For Integrated Risk

Why the Design Distribution Matters

In random-design regression, observations arrive where the covariate distribution puts them. A region with many observations can be estimated more precisely than a region with few observations, even if the regression function is equally smooth throughout the domain. Consequently, an integrated prediction risk depends not only on the regression function and noise, but also on the measure used to average prediction error.

Let the covariate \(X\) take values in \(\mathcal X=[0,1]^d\), with density \(f_X\) relative to Lebesgue measure. Write \(Y=m(X)+\varepsilon\), where \(m\) is the regression function and, for the examples below, the noise is independent of \(X\) with mean zero and variance \(\sigma^2\). Two common squared-error targets are the Lebesgue integrated risk and the design-weighted risk. The first weights locations equally per unit volume; the second weights them according to how often the design visits them.

R_{\mathrm{Leb}}(\hat m,m) = E\left[\int_{\mathcal X}\{\hat m(x)-m(x)\}^2\,dx\right], \qquad R_{\mathrm{des}}(\hat m,m) = E\left[\int_{\mathcal X}\{\hat m(x)-m(x)\}^2 f_X(x)\,dx\right].

The expectation is over the training data. The two risks ask different questions unless the design density is controlled. This tutorial explains the standard density conditions used to compare them, and shows what can break when those conditions fail. Such conditions support common upper- and lower-bound arguments; they are not a claim that every design must satisfy them.

Density Bounds and Risk Equivalence

Definition: The design density is bounded above and away from zero on \(\mathcal X\) if there are constants \(0<c\le C<\infty\) such that \(c\le f_X(x)\le C\) for Lebesgue-almost every \(x\in\mathcal X\).

The lower bound ensures that every region of positive volume receives design probability proportional to its volume. The upper bound rules out arbitrarily large design weights relative to volume. Together they make the two squared-error integrals comparable.

Theorem: Suppose \(\mathcal X\) has finite Lebesgue measure and \(c\le f_X\le C\) almost everywhere, for constants \(0<c\le C<\infty\). For any measurable estimator \(\hat m\) and measurable regression function \(m\), with risks allowed to be infinite, \(cR_{\mathrm{Leb}}(\hat m,m)\le R_{\mathrm{des}}(\hat m,m)\le CR_{\mathrm{Leb}}(\hat m,m)\).

Proof. For each realized training data set, the squared error \(g(x)=\{\hat m(x)-m(x)\}^2\) is nonnegative. Multiplying \(c\le f_X(x)\le C\) by \(g(x)\) and integrating gives

c\int_{\mathcal X}g(x)\,dx \le \int_{\mathcal X}g(x)f_X(x)\,dx \le C\int_{\mathcal X}g(x)\,dx.

Taking expectations preserves both inequalities for nonnegative random variables, including when an expectation is infinite. The result follows. \(\square\)

The theorem applies estimator by estimator, so it also compares minimax risks over the same parameter class. If \(\mathcal M\) is a class of regression functions, define \(R^*_{\mathrm{Leb}}=\inf_{\hat m}\sup_{m\in\mathcal M}R_{\mathrm{Leb}}(\hat m,m)\), and define \(R^*_{\mathrm{des}}\) analogously. Taking the supremum over \(m\) and then the infimum over estimators gives \(cR^*_{\mathrm{Leb}}\le R^*_{\mathrm{des}}\le CR^*_{\mathrm{Leb}}\). Thus rates agree up to constants. This is useful when a rate is derived for one integrated loss but the application uses the other.

Worked Example: Transferring a Risk Bound

Suppose \(X\) is uniform on \([0,1]\) with probability \(1/2\) and uniform on \([1,2]\) with probability \(1/2\). Its density is \(f_X(x)=1/2\) throughout \([0,2]\). An estimator has Lebesgue integrated risk \(R_{\mathrm{Leb}}=0.08\) on that domain. What is its design-weighted risk?

Since \(f_X(x)=1/2\), the design-weighted risk is exactly

R_{\mathrm{des}} = E\left[\int_0^2\{\hat m(x)-m(x)\}^2\frac12\,dx\right] = \frac12 R_{\mathrm{Leb}} = \frac12(0.08) = 0.04.

This density satisfies the condition with \(c=C=1/2\). The risk comparison gives \(0.5R_{\mathrm{Leb}}\le R_{\mathrm{des}}\le0.5R_{\mathrm{Leb}}\), which here is equality. If only the bounds \(1/4\le f_X\le1\) were known, the theorem would give \(0.02\le R_{\mathrm{des}}\le0.08\), not an exact value.

Why Lower Bounds Matter

The lower density bound has a direct interpretation for lower-bound arguments. If alternatives differ on a region of positive volume but the design never samples there, the data cannot distinguish them. In that case, estimation under Lebesgue integrated loss can be impossible, even when the alternatives are smooth. The next result makes this precise for a design distribution that assigns zero probability to part of the domain.

Theorem: Suppose two regression functions \(m_0,m_1\) induce exactly the same distribution for the observed data, and let \(D=\int_{\mathcal X}\{m_1(x)-m_0(x)\}^2\,dx<\infty\). For any estimator \(\hat m\), the average of its two Lebesgue risks satisfies \(\{R_{\mathrm{Leb}}(\hat m,m_0)+R_{\mathrm{Leb}}(\hat m,m_1)\}/2\ge D/4\).

Proof. For any real numbers \(a,u,v\), expanding squares gives \[ (a-u)^2+(a-v)^2 = 2\left(a-\frac{u+v}{2}\right)^2+\frac{(u-v)^2}{2} \ge \frac{(u-v)^2}{2}. \] Apply this pointwise with \(a=\hat m(x)\), \(u=m_0(x)\), and \(v=m_1(x)\), then integrate over \(x\). For any realized estimator, \[ \|\hat m-m_0\|_2^2+\|\hat m-m_1\|_2^2\ge D/2. \] Because the data distributions under the two functions are identical, the expectation of the left side under that common distribution equals the sum of the two risks. Dividing by two proves the stated average-risk bound. \(\square\)

Worked Example: An Unobserved Region Defeats Lebesgue-Risk Estimation

Let \(X\) have density \(f_X(x)=2\) on \([0,1/2]\) and zero on \((1/2,1]\). Take independent noise \(\varepsilon\sim N(0,1)\). Set \(m_0(x)=0\), and let \(m_1(x)=\delta\psi(x)\), where \(\delta>0\) and \(\psi\) is a nonzero infinitely differentiable function supported inside \((3/4,1)\).

Such a smooth bump can be constructed by translating and rescaling the function \(\eta(t)=\exp\{-1/[t(1-t)]\}\) for \(0<t<1\), extended by zero outside \((0,1)\). The extension is infinitely differentiable because all its derivatives approach zero at both endpoints. Its support lies where the design density is zero. Therefore \(m_0(X)=m_1(X)=0\) almost surely, and under both functions the observed response is just \(\varepsilon\). The distributions of the entire training data sets are identical for every sample size \(n\).

The squared Lebesgue separation is \[ D=\int_0^1\{\delta\psi(x)\}^2\,dx =\delta^2\int_0^1\psi(x)^2\,dx>0. \] The theorem implies that for every estimator its maximum risk over these two functions is at least their average risk, and hence at least \(D/4\). This bound does not shrink with \(n\). If a smoothness ball contains zero in its interior, \(\delta\) can be chosen small enough that both functions lie in the ball; the smooth bump construction works for arbitrarily high finite smoothness. The failure is not roughness. It is that the design supplies no information on a region counted by the target loss.

Degeneration Without a Completely Missing Region

A density can be positive almost everywhere and still fail to be bounded away from zero. Then exact indistinguishability need not occur, but the usual uniform comparison of the two risks is unavailable. Local sample information can vary sharply across the domain, so a bound derived under a uniform density condition cannot simply be transferred.

Worked Example: A Density That Vanishes at the Boundary

On \([0,1]\), take \(f_X(x)=2x\), which integrates to one but has infimum zero. Consider a smooth bump supported in \([0,h]\), of the form \(g_h(x)=h^s\psi(x/h)\), with \(\psi\) supported in \([0,1]\) and \(s>0\). Its Lebesgue squared norm is

\int_0^h g_h(x)^2\,dx = h^{2s+1}\int_0^1\psi(t)^2\,dt,

by the substitution \(t=x/h\). Its design-weighted squared norm is different:

\int_0^h g_h(x)^2\,2x\,dx = 2h^{2s+2}\int_0^1t\psi(t)^2\,dt.

For Gaussian noise of variance one, the Kullback–Leibler divergence between the \(n\)-observation laws under regression functions \(0\) and \(g_h\) is \[ \frac{n}{2}\int_0^1g_h(x)^2f_X(x)\,dx = nh^{2s+2}\int_0^1t\psi(t)^2\,dt. \] The factor \(h^{2s+2}\), rather than the Lebesgue scale \(h^{2s+1}\), reflects the low design mass near zero. This calculation does not by itself establish a global minimax rate: that requires accounting for all possible alternatives and the full parameter class. It does show why a uniform-density lower-bound calculation cannot be reused unchanged at this boundary.

What the Upper Density Bound Does—and Does Not—Say

The upper bound \(f_X\le C\) supplies the other direction in the risk comparison. Without it, a small region can receive a very large share of the design probability, and design-weighted error there can be large relative to its Lebesgue contribution. This does not mean that a high density is intrinsically harmful: more observations in a region can help estimate its regression function. It means only that the two integrated losses need not be uniformly comparable.

Worked Example: An Unbounded Density Breaks the Reverse Comparison

On \([0,1]\), let \(f_X(x)=1/(2\sqrt{x})\) for \(0<x\le1\). This is a density because \(\int_0^1(1/(2\sqrt{x}))\,dx=1\), but it is unbounded near zero. For the squared error function \(g_\epsilon(x)=\mathbf 1_{[0,\epsilon]}(x)\), where \(0<\epsilon\le1\), the Lebesgue and design-weighted integrals are

\int_0^1g_\epsilon(x)\,dx=\epsilon, \qquad \int_0^1g_\epsilon(x)f_X(x)\,dx = \int_0^\epsilon\frac{dx}{2\sqrt{x}} = \sqrt{\epsilon}.

Their ratio is \(1/\sqrt{\epsilon}\), which becomes arbitrarily large as \(\epsilon\) decreases. Thus no finite constant \(C\) can make the design-weighted integral at most \(C\) times the Lebesgue integral for all nonnegative error functions. The density upper bound is what rules out this failure of uniform comparison.

Using Density Conditions Carefully

Bounds above and away from zero are particularly convenient when the target is Lebesgue integrated risk and one wants design-based calculations to control that target uniformly. They also keep lower-bound constructions from becoming artificially easy or hard simply because a chosen region receives vanishing or extreme design mass. But the conditions are sufficient tools, not universal prerequisites for every rate result. If the target is design-weighted risk, that weighting may be exactly the intended loss. If the density degenerates, rates may still be available, but they can depend on how the density behaves and on where the loss places weight.

A common mistake is to write “the design density is positive” and infer risk equivalence. Positivity almost everywhere is not enough: \(f_X(x)=2x\) is positive almost everywhere on \([0,1]\) but has no positive lower bound, and the boundary calculation above shows the resulting change in local information. Conversely, a density that is bounded below but not above gives only one direction of comparison, not both. State the actual bounds and the target measure before transferring an upper or lower risk bound.

Key takeaway: If \(c\le f_X\le C\), then Lebesgue and design-weighted integrated squared risks are equivalent up to fixed constants. If the density approaches zero or becomes unbounded, that equivalence can fail; identify the affected regions and derive the risk argument for the actual design rather than assuming the usual comparison.

Check Your Understanding

Use the definitions and comparisons above to answer the following.

For \(f_X(x)=3/2\) on \([0,2/3]\) and zero elsewhere in \([0,1]\), which part of the risk-equivalence condition fails for Lebesgue risk on \([0,1]\), and why?
If \(1/5\le f_X(x)\le 4\) almost everywhere, state both inequalities relating \(R_{\mathrm{des}}\) and \(R_{\mathrm{Leb}}\).
Why can two regression functions that differ only on a zero-design-probability region induce identical data distributions but have positive Lebesgue separation?
For \(f_X(x)=2x\), compare the powers of \(h\) in the two squared-norm calculations for \(g_h(x)=h^s\psi(x/h)\).
Does an unbounded density necessarily make prediction harder? Explain what the unboundedness does establish about uniform equivalence of the two risks.