Density Estimation on Unbounded Supports

Why Unbounded Supports Require a Tail Analysis

Many densities are defined on the whole real line or on an unbounded subset of Euclidean space. An estimator may nevertheless be computed only on a finite region, either because the method uses a finite basis or because evaluation outside a chosen range is impractical. Restricting the estimate in this way creates an error that does not disappear merely because the estimator is accurate where it is computed: the true density may still have substantial mass in the omitted tail.

For integrated squared error, the relevant omitted quantity is the integral of the squared density over that tail, not its probability mass. The two quantities can decay at different rates. This distinction matters when translating a tail assumption into a risk bound and when deciding how large the estimation region should be.

Truncation and Its Exact Risk Cost

Let $X_1,\ldots,X_n$ be independent observations from a density $f$ on $\mathbb{R}^d$. For a region $A_T$, such as the cube $[-T,T]^d$, write $A_T^c$ for its complement. Start with an estimator $\tilde f$ and set it to zero outside the region. This gives the truncated estimator $\hat f_T(x)=\tilde f(x)\mathbf{1}\{x\in A_T\}$. It need not integrate to one; here it is used as an estimator under integrated squared error.

Definition: The integrated squared-error risk of an estimator $\hat f$ is $E\int_{\mathbb{R}^d}(\hat f(x)-f(x))^2\,dx$, when this expectation is finite. The squared-density tail beyond $A_T$ is $\tau_2(T)=\int_{A_T^c}f(x)^2\,dx$.

If $\tilde f$ is square-integrable on $A_T$, the truncation error splits exactly into the error within the region and the tail contribution outside it.

Theorem: For any measurable estimator $\tilde f$ for which the displayed risks are defined, and $\hat f_T=\tilde f\mathbf{1}\{x\in A_T\}$, $$ E\int_{\mathbb{R}^d}(\hat f_T-f)^2 = E\int_{A_T}(\tilde f-f)^2+\int_{A_T^c}f^2. $$ Thus $\tau_2(T)$ is an unavoidable contribution to integrated squared error for any estimator that is zero outside $A_T$.

To prove the identity, partition the integral over $A_T$ and $A_T^c$. On $A_T$, $\hat f_T=\tilde f$; on $A_T^c$, $\hat f_T=0$. Therefore, for each realized estimator, $$ \int_{\mathbb{R}^d}(\hat f_T-f)^2 = \int_{A_T}(\tilde f-f)^2+\int_{A_T^c}f^2. $$ The second term is nonrandom. Taking expectations gives the result. The integrands are nonnegative, so the expectation and integrals are well-defined as nonnegative extended integrals; finiteness of the displayed risk makes the identity an equality of finite quantities. In particular, the first term cannot be negative, so the risk is at least $\tau_2(T)$. $\square$

Worked Example: Exact Tail Error for a Pareto Density

Let $f(x)=x^{-2}$ for $x\geq 1$, and $f(x)=0$ otherwise. This is a density because $\int_1^\infty x^{-2}\,dx=1$. Suppose an estimator is set to zero for $x>T$, where $T\geq1$.

The omitted probability mass is $$ P(X>T)=\int_T^\infty x^{-2}\,dx=\frac{1}{T}. $$ The omitted contribution to integrated squared error is instead $$ \tau_2(T)=\int_T^\infty (x^{-2})^2\,dx =\int_T^\infty x^{-4}\,dx =\left[-\frac{1}{3x^3}\right]_T^\infty =\frac{1}{3T^3}. $$

At $T=10$, the estimator omits probability mass $0.1$, but the unavoidable squared-error contribution is $1/3000$, approximately $0.000333$. They are different measures of tail size; neither can be substituted for the other without checking which loss is being analyzed.

How Tail Assumptions Bound Integrated Error

A useful way to turn a qualitative statement such as “the density has a polynomial tail” into a risk bound is to assume a pointwise envelope. Let $\|\cdot\|$ denote Euclidean norm and let $s_{d-1}$ be the surface area of the unit sphere in $\mathbb{R}^d$. The next result states the condition and the resulting squared-tail rate.

Theorem: Suppose that for some $C>0$ and $\alpha>d$, $f(x)\leq C\|x\|^{-\alpha}$ for almost every $x$ with $\|x\|\geq1$. For $T\geq1$, let $A_T=\{x:\|x\|\leq T\}$. Then $$ \int_{A_T^c} f(x)^2\,dx \leq \frac{C^2s_{d-1}}{2\alpha-d}\,T^{d-2\alpha}. $$ The same envelope also gives $P(X\notin A_T)\leq Cs_{d-1}T^{d-\alpha}/(\alpha-d)$.

For the first inequality, square the assumed envelope and integrate it over the region $\|x\|>T$. The polar-coordinate formula for a nonnegative radial function gives $$ \int_{\|x\|>T} f(x)^2\,dx \leq C^2s_{d-1}\int_T^\infty r^{d-1-2\alpha}\,dr. $$ Because $\alpha>d$, the exponent $d-1-2\alpha$ is less than $-1$. Hence the integral converges and equals $$ \int_T^\infty r^{d-1-2\alpha}\,dr =\frac{T^{d-2\alpha}}{2\alpha-d}. $$ This proves the squared-tail bound. For the probability tail, apply the same argument to $f$, rather than $f^2$: $$ P(X\notin A_T) \leq Cs_{d-1}\int_T^\infty r^{d-1-\alpha}\,dr =\frac{Cs_{d-1}}{\alpha-d}T^{d-\alpha}. $$ The condition $\alpha>d$ ensures convergence of both integrals. $\square$

The exponents differ: the probability tail is bounded at order $T^{d-\alpha}$, while the squared-density tail is bounded at order $T^{d-2\alpha}$. In one dimension with a density proportional to $x^{-2}$ on the positive tail, the probability tail decays as $T^{-1}$ and the squared tail as $T^{-3}$. The latter is the quantity that enters the exact integrated squared-error decomposition.

Worked Example: Gaussian Tail Contribution to Squared Error

Let $f$ be the $N(0,1)$ density, $f(x)=(2\pi)^{-1/2}\exp(-x^2/2)$, and truncate to $[-T,T]$. Since $f(x)^2=(2\pi)^{-1}\exp(-x^2)$, the squared-density tail is $$ \tau_2(T)=\frac{1}{2\pi}\int_{|x|>T}e^{-x^2}\,dx =\frac{1}{2\sqrt{\pi}}\operatorname{erfc}(T), $$ where $\operatorname{erfc}(T)=(2/\sqrt{\pi})\int_T^\infty e^{-u^2}\,du$.

At $T=3$, $\operatorname{erfc}(3)$ is approximately $2.21\times10^{-5}$, so $\tau_2(3)$ is approximately $6.23\times10^{-6}$. By contrast, the omitted probability mass is $P(|X|>3)$, approximately $0.00270$. The mass is much larger than the squared-error tail in this example. A risk calculation should use the integral dictated by its loss, rather than infer squared error directly from probability mass.

Choosing the Estimation Region

A larger region reduces the omitted tail but can increase the cost of estimating within the region. The precise balance depends on the estimator: for some kernel estimators the integrated variance can be bounded over the entire line without a factor proportional to region size, while methods that fit a number of local components growing with the region may have a size-dependent variance cost. Thus a region-selection rule must use a risk bound appropriate to the method, not assume that every estimator has the same variance behavior.

For a concrete tradeoff, suppose a one-dimensional procedure has an interior risk bound of the form $$ E\int_{-T}^T(\tilde f-f)^2 \leq C_bh^{2\beta}+\frac{C_vT}{nh}, $$ where $h>0$ is a resolution parameter, $\beta>0$ is a smoothness index, and $C_b,C_v>0$ do not depend on $T,n,h$ over the range considered. Such a bound describes a method whose estimation cost grows with the length of the region. If also $\tau_2(T)\leq C_tT^{-q}$ for $q>0$, the exact decomposition gives $$ E\int_{\mathbb{R}}(\hat f_T-f)^2 \leq C_bh^{2\beta}+\frac{C_vT}{nh}+C_tT^{-q}. $$ For fixed $h$, choosing $T$ means balancing the last two terms. The smoothness term does not depend on $T$ under the stated bound.

Worked Example: Balancing Variance and a Polynomial Tail

Use the density $f(x)=x^{-2}$ for $x\geq1$ from the earlier calculation. For $T\geq1$, its squared tail is exactly $1/(3T^3)$. Suppose the interior bound has variance term $T/(nh)$. Ignoring a bias term that is constant in $T$, the bound to minimize is $$ R(T)=\frac{T}{nh}+\frac{1}{3T^3}. $$

Differentiate for $T>0$: $$ R'(T)=\frac{1}{nh}-T^{-4}. $$ The stationary point solves $T^4=nh$, giving $T_*=(nh)^{1/4}$. The second derivative is $R''(T)=4T^{-5}>0$, so this stationary point is a minimum. It is an admissible choice in the stated range when $nh\geq1$. At that choice, $$ \frac{T_*}{nh}=(nh)^{-3/4}, \qquad \frac{1}{3T_*^3}=\frac{1}{3}(nh)^{-3/4}, $$ and therefore $R(T_*)=(4/3)(nh)^{-3/4}$. The example shows how the tail exponent affects the region choice: a slower-decaying squared tail would require a larger region to make truncation comparably small.

Truncation Versus Renormalization

Setting the estimate to zero outside the region generally produces a function whose integral is less than one, so it is not itself a density estimate in the strict sense. Renormalization can restore total integral one, but it changes the error: it inflates the estimate inside the region and does not recover the missing shape of the true tail.

This can be seen without sampling error by considering the population density conditional on $X\in A_T$. Let $\delta=P(X\notin A_T)$, with $0\leq\delta<1$, and define $f_T(x)=f(x)\mathbf{1}\{x\in A_T\}/(1-\delta)$. This is a density. Its integrated squared distance from $f$ is $$ \int_{\mathbb{R}^d}(f_T-f)^2 = \frac{\delta^2}{(1-\delta)^2}\int_{A_T}f^2 +\int_{A_T^c}f^2. $$ Indeed, on $A_T$, the difference is $\delta f/(1-\delta)$; outside, it is $-f$. Squaring and integrating over these disjoint regions gives the formula. The tail term remains, and renormalization adds an inside distortion. If $1-\delta$ is small, the multiplier $\delta^2/(1-\delta)^2$ can be large. In a data-based procedure, the random normalizing integral can also be close to zero, so its behavior must be controlled before using a risk bound.

Common Mistake

Incorrect reasoning: “The probability outside the region is at most $\epsilon$, so the integrated squared-error contribution from truncation is at most $\epsilon$.” This treats probability tail and squared-density tail as interchangeable.

Correction: For an estimator set to zero outside $A_T$, the exact truncation contribution under integrated squared error is $\int_{A_T^c}f^2$. A bound on $P(X\notin A_T)=\int_{A_T^c}f$ alone does not give the desired bound without additional information about the size or shape of $f$ in that tail. State a suitable tail envelope or another condition that directly controls the squared-density integral, and use the integral corresponding to the loss being analyzed.

Key takeaway: Truncation creates an exact squared-tail term in integrated squared error. Tail assumptions quantify that term, while the estimator’s own risk bound determines whether a larger region is worth its additional estimation cost.

Check Your Understanding

Use the definitions and risk decomposition above to answer the following.

For an estimator that is zero outside $A_T$, what exact integral gives its unavoidable contribution to integrated squared error?
Under the envelope $f(x)\leq C\|x\|^{-\alpha}$ in dimension $d$, why is $\alpha>d$ sufficient for the probability-tail integral to converge?
Why can a small probability tail not be substituted directly for a small squared-density tail?
If a method’s variance cost increases with the size of its estimation region, what two terms should generally be balanced when choosing that region?
What additional source of error can arise when a truncated density is renormalized?