Uniform Tail Bounds for Truncated Risk

Why Uniform Tail Control Matters

For an unbounded loss, replacing the loss by a capped version makes the risk easier to control, but it changes the decision problem. The key question is not merely whether the truncated risk approaches the original risk for each fixed parameter and rule. In minimax work, both may vary: a useful approximation must control the discrepancy uniformly over the parameters and rules under consideration.

The relevant condition is uniform integrability of the family of losses. It prevents small-probability events with very large losses from carrying a substantial amount of expected loss. This tutorial develops the precise tail bound and shows how moment assumptions can provide it.

Losses, Risks, and Uniform Integrability

Let $X$ have distribution $P_\theta$ when the parameter is $\theta\in\Theta$, and let $\mathcal D$ be a collection of measurable decision rules. Suppose the loss $\ell(\theta,\delta(X))$ is nonnegative and measurable for every $\theta\in\Theta$ and $\delta\in\mathcal D$. Write

W_{\theta,\delta}=\ell(\theta,\delta(X)),\qquad R(\theta,\delta)=E_\theta[W_{\theta,\delta}].

For a truncation level $c>0$, define the truncated loss and its risk by

W_{\theta,\delta}^{(c)}=\min(W_{\theta,\delta},c),\qquad R_c(\theta,\delta)=E_\theta[W_{\theta,\delta}^{(c)}].

Expectations here may initially be understood as nonnegative extended expectations. The family to examine is the collection of distributions of $W_{\theta,\delta}$ as both $\theta$ and $\delta$ vary.

Definition: The nonnegative family $\{W_{\theta,\delta}:\theta\in\Theta,\ \delta\in\mathcal D\}$ is uniformly integrable if $$ \lim_{c\to\infty}\ \sup_{\theta\in\Theta,\,\delta\in\mathcal D} E_\theta\!\left[W_{\theta,\delta}\mathbf 1\{W_{\theta,\delta}>c\}\right]=0. $$ This tail-expectation criterion is the condition used below. It requires the expected contribution from large losses to vanish uniformly, not just separately for each fixed pair $(\theta,\delta)$.

A useful identity explains why this definition fits truncation. For every nonnegative number $w$, $\,w-\min(w,c)=(w-c)_+$, where $x_+=\max(x,0)$. Consequently,

R(\theta,\delta)-R_c(\theta,\delta) =E_\theta[(W_{\theta,\delta}-c)_+].

The difference is nonnegative. Also, $(w-c)_+\leq w\mathbf 1\{w>c\}$, so the tail expectation in the definition directly bounds the truncation error.

Worked Example: A Uniform Tail Bound for Gaussian Estimation

Suppose $X\sim N(\theta,1)$, where $\theta\in\mathbb R$, and consider the rules $\delta_t(X)=X+t$ for $-1\leq t\leq1$. Use squared-error loss, so $W_{\theta,t}=(\delta_t(X)-\theta)^2$. Find a bound on the truncation error that holds for every $\theta$ and $t$.

Write $X=\theta+Z$, where $Z\sim N(0,1)$. Then $W_{\theta,t}=(Z+t)^2$, and its squared moment is

E_\theta[W_{\theta,t}^2] =E[(Z+t)^4] =E[Z^4]+4tE[Z^3]+6t^2E[Z^2]+4t^3E[Z]+t^4 =3+6t^2+t^4.

For $|t|\leq1$, this is at most $3+6+1=10$, independently of $\theta$. On the event $W_{\theta,t}>c$, we have $W_{\theta,t}\leq W_{\theta,t}^2/c$. Therefore

0\leq R(\theta,\delta_t)-R_c(\theta,\delta_t) \leq E_\theta[W_{\theta,t}\mathbf 1\{W_{\theta,t}>c\}] \leq \frac{E_\theta[W_{\theta,t}^2]}{c} \leq \frac{10}{c}.

Thus the discrepancy is at most $0.1$ for every parameter and every rule in this collection when $c=100$. The bound does not depend on $\theta$, even though the parameter space is unbounded.

The Uniform Truncation Bound

Theorem: Suppose the losses $W_{\theta,\delta}$ are nonnegative and the family is uniformly integrable. Define $$ \varepsilon(c)= \sup_{\theta\in\Theta,\,\delta\in\mathcal D} E_\theta[W_{\theta,\delta}\mathbf 1\{W_{\theta,\delta}>c\}]. $$ Then, for every $c>0$, $$ \sup_{\theta\in\Theta,\,\delta\in\mathcal D} |R(\theta,\delta)-R_c(\theta,\delta)| \leq \varepsilon(c), $$ and $\varepsilon(c)\to0$ as $c\to\infty$. In particular, all the risks in this family are finite for sufficiently large $c$, and the truncation error tends to zero uniformly.

Proof. For each $(\theta,\delta)$, the identity above gives $R(\theta,\delta)-R_c(\theta,\delta)=E_\theta[(W_{\theta,\delta}-c)_+]$. Pointwise, \[ 0\leq (W_{\theta,\delta}-c)_+ \leq W_{\theta,\delta}\mathbf 1\{W_{\theta,\delta}>c\}. \] Taking expectations and then the supremum gives the claimed inequality. Uniform integrability says precisely that its right-hand side tends to zero. To check finiteness, choose a sufficiently large $c$ for which $\varepsilon(c)<\infty$. Since $W\leq c+W\mathbf 1\{W>c\}$, every expectation in the family is at most $c+\varepsilon(c)$. Thus the original risks are finite as well. $\square$

The finiteness conclusion is useful: the tail condition does not merely compare finite quantities that have been assumed in advance. It also bounds the risks uniformly once the tail expectation is finite at some truncation level.

A Moment Condition That Guarantees Uniformity

Uniform integrability can be established by bounding a moment strictly higher than the first. This gives a quantitative rate for the truncation error.

Theorem: Let $q>1$. If for some finite $C$, $$ \sup_{\theta\in\Theta,\,\delta\in\mathcal D} E_\theta[W_{\theta,\delta}^{\,q}]\leq C, $$ then the family is uniformly integrable and $$ \sup_{\theta,\delta}|R(\theta,\delta)-R_c(\theta,\delta)| \leq \frac{C}{c^{q-1}}. $$

Proof. When $W>c$, $W=W^q/W^{q-1}\leq W^q/c^{q-1}$. Hence

E_\theta[W_{\theta,\delta}\mathbf 1\{W_{\theta,\delta}>c\}] \leq \frac{E_\theta[W_{\theta,\delta}^{\,q}]}{c^{q-1}} \leq \frac{C}{c^{q-1}}.

Taking the supremum shows that the tail expectation tends to zero. Applying the uniform truncation bound proves the stated risk inequality. $\square$

Worked Example: Bounded Loss Gives Exact Truncation

Let $X\sim\operatorname{Bernoulli}(\theta)$, with $0\leq\theta\leq1$, and let every rule take an action $a\in[0,1]$. Under squared-error loss for estimating $\theta$, the loss is $W=(a-\theta)^2$.

Since both $a$ and $\theta$ lie in $[0,1]$, $|a-\theta|\leq1$, and therefore $0\leq W\leq1$. For any $c\geq1$, $\min(W,c)=W$ for every possible outcome. It follows directly that

R_c(\theta,\delta)=R(\theta,\delta) \qquad\text{for every }\theta,\delta\text{ and }c\geq1.

Here no tail estimate is needed: the losses have a common finite upper bound. This is the strongest possible form of uniform approximation.

Worked Example: Uniform Integrability Without a Bounded Loss

Return to the Gaussian rules $\delta_t(X)=X+t$ with $|t|\leq1$ and squared-error loss. The loss is not bounded: a normal observation can be arbitrarily far from $\theta$. Nevertheless, the fourth-moment calculation gave $\sup_{\theta,t}E_\theta[W_{\theta,t}^2]\leq10$. Apply the moment theorem with $q=2$ and $C=10$.

For every $c>0$, it yields

\sup_{\theta\in\mathbb R,\,|t|\leq1} |R(\theta,\delta_t)-R_c(\theta,\delta_t)| \leq \frac{10}{c}.

The moment argument supplies a simple, conservative bound; it does not claim that the actual discrepancy equals $10/c$. Its value is that a single calculation verifies control over the whole parameter and rule collection.

Why Pointwise Convergence Is Not Enough

For each fixed nonnegative integrable loss $W$, the truncated risk approaches its risk as $c$ increases. But this pointwise fact does not control a supremum over parameters and rules. A family can put progressively smaller probabilities on progressively larger losses, keeping the expected tail contribution large.

Worked Example: Pointwise Convergence With No Uniform Approximation

Let the parameter space be the positive integers, use a single decision rule, and let the observation $X$ take values in $\{0,1,2,\ldots\}$. Under parameter $\theta=j$, define $P_j(X=j)=1/j$ and $P_j(X=0)=1-1/j$. Use the same nonnegative loss $\ell(X)=X$ for every parameter.

At each fixed $j$, the risk is $R(j)=E_j[X]=j(1/j)=1$. Its truncated version is

R_c(j)=E_j[\min(X,c)] =\frac{\min(j,c)}{j}.

For any fixed $j$, once $c\geq j$, $R_c(j)=1$. Thus $R_c(j)\to R(j)$ for every fixed parameter. But if $j>c$, the truncation error is

R(j)-R_c(j)=1-\frac{c}{j}.

For every fixed $c$, this error approaches $1$ as the integer $j$ increases. Its supremum over $j$ is therefore $1$, not a quantity that vanishes with $c$. In fact, the family is not uniformly integrable: for integer $j>c$, $E_j[X\mathbf 1\{X>c\}]=1$. Pointwise convergence alone misses exactly the tail behavior that matters for uniform approximation.

Common Pitfall and Practical Use

A common mistake is to show that $R_c(\theta,\delta)\to R(\theta,\delta)$ for each fixed $(\theta,\delta)$, then conclude that the convergence is uniform. The example above disproves that inference. The correction is to bound the tail expectation with a supremum taken over the entire candidate family:

\sup_{\theta,\delta} E_\theta[W_{\theta,\delta}\mathbf 1\{W_{\theta,\delta}>c\}] \longrightarrow 0.

In applications, this condition may be verified directly or deduced from a common $q$-th moment bound for some $q>1$. If a common deterministic bound on the loss is available, truncation is exact above that bound. These checks tell you whether a truncated-risk approximation is valid uniformly over the parameter values and rules being compared. They do not by themselves establish that an optimal rule exists or that a sequence of rules converges; they control the approximation error in risk.

Check Your Understanding

Use the definitions and bounds from this tutorial to answer the following questions.

For a nonnegative loss $W$, express $E[W]-E[\min(W,c)]$ using the positive-part function.
Why does a uniform bound on $E[W^q]$ for $q>1$ imply uniform integrability?
If $\sup_{\theta,\delta}E_\theta[W_{\theta,\delta}^2]\leq 8$, what truncation-error bound follows at level $c$?
In the integer-parameter example, why does convergence at every fixed parameter fail to imply a vanishing supremum error?
What does uniform integrability establish about truncated risks, and what does it not establish about optimal rules?