Bounded Loss and Uniform Risk Bounds

Why Bounded Loss Helps in Complete-Class Arguments

A complete-class argument compares decision rules through their risk functions, often over many parameter values at once. To replace a rule by an approximation, it is not enough that the replacement behave well at a single parameter value: the risk error must be controlled across the whole parameter space. A common bound on the loss provides a direct way to obtain that control. If two rules differ only slightly in how they distribute probability over actions, bounded loss limits how much their risks can differ.

The bound is useful because it does not require the sampling distribution to have a density, or the parameter space to be finite. It does require the loss bound to hold for every parameter and relevant observation-action pair. The resulting risk guarantee is uniform only when the approximation error in the rules is itself uniform in the appropriate sense.

Bounded Loss and Total Variation

For probability measures \(Q_1,Q_2\) on the same measurable space, use the convention \(d_{\mathrm{TV}}(Q_1,Q_2)=\sup_A|Q_1(A)-Q_2(A)|\). With this convention, total variation lies between zero and one. The width \(C\), rather than the absolute size of the loss, determines the bound on a difference of expectations.

Proof. If \(M=m\), then \(g\) is constant and the difference is zero. Otherwise let \(h=(g-m)/(M-m)\), so \(0\le h\le1\). For any probability measure \(Q\), Tonelli's theorem applied to the nonnegative indicator \(\mathbf{1}\{h>t\}\) gives \(\int h\,dQ=\int_0^1 Q(h>t)\,dt\). Therefore \[ \left|\int h\,dQ_1-\int h\,dQ_2\right| \le \int_0^1 |Q_1(h>t)-Q_2(h>t)|\,dt \le d_{\mathrm{TV}}(Q_1,Q_2). \] Multiplying by \(M-m\) proves the result. \(\square\)

A Uniform Risk Perturbation Bound

Apply the lemma to the conditional distribution over actions at each observation. The sampling distribution \(P_\theta\) is kept fixed; only the decision rule changes. The measurable spaces and kernels below are understood to make the displayed integrals well-defined.

Proof. For fixed \(\theta\) and \(x\), the function of action \(a\mapsto L(\theta,a)\) takes values in \([m,M]\). The lemma, with the two conditional action distributions, implies \[ \left|\int L(\theta,a)\,\delta(da\mid x)-\int L(\theta,a)\,\tilde\delta(da\mid x)\right| \le C\,d_{\mathrm{TV}}\bigl(\delta(\,\cdot\mid x),\tilde\delta(\,\cdot\mid x)\bigr). \] Integrate with respect to \(P_\theta(dx)\). The absolute value of an integral is no greater than the integral of the absolute value, so the claimed inequality follows. If the integrated total variation is at most \(\varepsilon\) for every \(\theta\), the inequality holds with right side \(C\varepsilon\) for every \(\theta\); taking the supremum proves the uniform bound. \(\square\)

A convenient sufficient condition is a pointwise bound \(d_{\mathrm{TV}}(\delta(\,\cdot\mid x),\tilde\delta(\,\cdot\mid x))\le\varepsilon\) for every observation \(x\). Since \(P_\theta\) is a probability measure, its integral is then at most \(\varepsilon\) for every \(\theta\). This condition is stronger than necessary: the theorem only requires a uniform bound on the average discrepancy under each \(P_\theta\).

Using the Bound to Preserve Risk Comparisons

Uniform risk control is particularly useful when a complete-class argument constructs or approximates rules. Suppose \(\delta\) has a strict risk advantage over another rule \(\eta\), with a common margin across parameters. A sufficiently accurate approximation to \(\delta\) retains that advantage. Without a common margin, a small uniform error guarantees approximate comparison, but need not preserve exact domination.

Proof. For every \(\theta\), the uniform error bound gives \(R(\theta,\tilde\delta)\le R(\theta,\delta)+\epsilon\). The strict-margin assumption then yields \[ R(\theta,\tilde\delta) \le R(\theta,\eta)-\gamma+\epsilon < R(\theta,\eta), \] because \(\epsilon<\gamma\). This holds at every parameter value. \(\square\)

The margin condition matters. If all that is known is \(R(\theta,\delta)\le R(\theta,\eta)\), an approximation error of size \(\epsilon\) only yields \(R(\theta,\tilde\delta)\le R(\theta,\eta)+\epsilon\); it does not establish exact domination. In Wald's compactness-based complete-class argument, bounded loss can supply the uniform risk control needed when comparing rules, but boundedness alone does not establish every other condition in such an argument.

Common Pitfall: Pointwise Control Is Not Uniform Control

A frequent error is to show that a risk difference becomes small for each fixed \(\theta\), and then conclude that its supremum over \(\theta\) is small. Those are different claims. For example, on parameter space \([0,1]\), define \(r_n(\theta)=\mathbf{1}\{\theta\le1/n\}\). For every fixed \(\theta>0\), \(r_n(\theta)\) is eventually zero, and at \(\theta=0\) it is always one; so this particular sequence does not converge pointwise to zero at every parameter. A modified example makes the distinction exact: set \(r_n(\theta)=\mathbf{1}\{\theta=1/n\}\). Then for every fixed \(\theta\), \(r_n(\theta)\) is eventually zero, but \(\sup_{\theta\in[0,1]}r_n(\theta)=1\) for every \(n\). Pointwise convergence by itself cannot yield a uniform bound.

The correction is to establish the bound required by the theorem directly: for instance, control the total variation discrepancy uniformly in \(x\), or prove that its expectation under \(P_\theta\) is bounded by the same \(\varepsilon\) for every \(\theta\). Also check the loss range. If losses can grow without bound across actions or parameter values, a small total variation discrepancy does not by itself provide the stated finite uniform risk bound.