Noncompact Parameters and Failed Risk Approximation

Why Pointwise Risk Approximation Can Mislead

A risk function assigns a performance value to every parameter value. When the parameter space is noncompact, a sequence of rules can behave increasingly well at each fixed parameter while continuing to perform poorly somewhere else. The location of that poor performance may move farther out through the parameter space as the sequence advances.

This distinction matters in class-reduction arguments. A pointwise limit of risks need not be uniformly close to any rule in the approximating class. In particular, pointwise approximation alone does not establish the uniform additive bound required for an epsilon-complete replacement.

Definition: In a decision problem with parameter $\theta$, observation $X$, action $a$, and loss $L(\theta,a)$, the risk of a rule $\delta$ is $$ R(\theta,\delta)=E_\theta[L(\theta,\delta(X))]. $$ A sequence of rules $\delta_m$ has pointwise risk convergence to a function $r$ if, for every fixed $\theta$, $R(\theta,\delta_m)\to r(\theta)$. It has uniform risk convergence if $$ \sup_{\theta\in\Theta}|R(\theta,\delta_m)-r(\theta)|\longrightarrow 0. $$ Uniform convergence controls the error over all parameter values at once; pointwise convergence only controls it at each fixed value.

The supremum in the uniform definition is especially important when a replacement claim must hold for every parameter. A sequence may give excellent approximations on any fixed bounded region, while its worst error over the entire parameter space never decreases.

A Moving-Spike Decision Problem

Let the parameter space be $\Theta=\mathbb{R}$, which is noncompact with its usual topology. The observation is $X=\theta$ with probability one under $P_\theta$. Thus the parameter is observed exactly. Let the action space be $\{0,1\}$, with loss $L(\theta,a)=a$. For each positive integer $m$, define the measurable rule $\delta_m(x)=1$ if $x=m$, and $\delta_m(x)=0$ otherwise. Also define $\delta_0(x)=0$ for every $x$.

Worked Example: Pointwise Convergence With a Persistent Worst-Case Error

Compute the risks of $\delta_m$ and $\delta_0$, then compare pointwise and uniform convergence.

Because $X=\theta$ with probability one and the loss is the action itself, $$ R(\theta,\delta_m)=E_\theta[\delta_m(X)]=\delta_m(\theta) =\begin{cases} 1,&\theta=m,\\ 0,&\theta\ne m, \end{cases} \qquad R(\theta,\delta_0)=0. $$

Fix any $\theta\in\mathbb{R}$. If $\theta$ is not a positive integer, then $R(\theta,\delta_m)=0$ for every $m$. If $\theta=k$ is a positive integer, then the risk is 1 only when $m=k$, and is 0 for all $m\ne k$. In either case, $$ R(\theta,\delta_m)\longrightarrow 0=R(\theta,\delta_0). $$ The convergence is pointwise.

For every $m$, however, taking $\theta=m$ gives $$ \sup_{\theta\in\mathbb{R}}|R(\theta,\delta_m)-R(\theta,\delta_0)|=1. $$ So the risks do not converge uniformly. The error has not disappeared; it has moved to the parameter value $\theta=m$, which changes with the rule.

The noncompactness is visible in the movement of these exceptional parameter values. Every fixed $\theta$ is eventually left behind, but the whole parameter space always contains the current exceptional value. Pointwise convergence records the first fact and does not record the second.

What Uniform Convergence Would Guarantee

Uniform control is useful because it controls worst-case risk. The following elementary result makes the distinction precise. Assume the risk functions are bounded, so their suprema are finite.

Theorem: If $r_m$ and $r$ are bounded real-valued functions on $\Theta$ and $\sup_{\theta\in\Theta}|r_m(\theta)-r(\theta)|\to 0$, then $$ \sup_{\theta\in\Theta}r_m(\theta)\longrightarrow \sup_{\theta\in\Theta}r(\theta). $$

Proof. Put $d_m=\sup_{\theta\in\Theta}|r_m(\theta)-r(\theta)|$. For every $\theta$, $r_m(\theta)\le r(\theta)+d_m$, so taking suprema gives $\sup_\theta r_m(\theta)\le\sup_\theta r(\theta)+d_m$. Interchanging $r_m$ and $r$ gives $\sup_\theta r(\theta)\le\sup_\theta r_m(\theta)+d_m$. Therefore $$ \left|\sup_{\theta\in\Theta}r_m(\theta)-\sup_{\theta\in\Theta}r(\theta)\right|\le d_m\longrightarrow 0. $$ This proves the claim. $\square$

Pointwise convergence does not give the bound $d_m\to0$. In the moving-spike example, $\sup_\theta R(\theta,\delta_m)=1$ for every $m$, whereas $\sup_\theta R(\theta,\delta_0)=0$. Thus the worst-case risks do not converge to the worst-case risk of the pointwise limit.

Worked Example: A Weighted Distance Can Shrink While the Worst Error Does Not

Restrict attention to the countable parameter subset $\Theta=\{1,2,\ldots\}$, and assign weight $w_j=2^{-j}$ to parameter $j$. The weights are positive and sum to 1. Consider the weighted distance between two risk functions $r$ and $s$, $$ d_w(r,s)=\sum_{j=1}^{\infty}2^{-j}|r(j)-s(j)|. $$

On this parameter set, the risk of $\delta_m$ is 1 at $j=m$ and 0 at every other $j$; the risk of $\delta_0$ is identically zero. Hence only one term in the distance is nonzero: $$ d_w(R(\cdot,\delta_m),R(\cdot,\delta_0)) =2^{-m}|1-0|=2^{-m}\longrightarrow 0. $$ Yet the uniform distance remains $$ \sup_{j\ge1}|R(j,\delta_m)-R(j,\delta_0)|=1 $$ for every $m$. A weighted distance that gives small weight to distant indices can make the moving spike cheap, but it does not turn the approximation into uniform control.

Consequences for Replacing Rules

A class $C$ is epsilon-complete, under the usual additive-risk convention, if for every rule $\delta$ and every $\varepsilon>0$, there is some $\eta\in C$ such that $R(\theta,\eta)\le R(\theta,\delta)+\varepsilon$ for every $\theta$. The “for every $\theta$” clause is a simultaneous bound. Showing that risks from $C$ converge pointwise to the risk of $\delta$ does not by itself supply a member $\eta$ satisfying that clause.

Worked Example: Pointwise Risk Density Does Not Give Epsilon-Completeness

Let $C=\{\delta_m:m=1,2,\ldots\}$ in the moving-spike problem, and take $\delta_0$ as the rule to be replaced. The risks of members of $C$ converge pointwise to $R(\cdot,\delta_0)$. Nevertheless, no member of $C$ is within a uniform additive error less than 1 of $\delta_0$.

Indeed, for any proposed replacement $\delta_k\in C$, evaluate the risks at the parameter value $\theta=k$: $$ R(k,\delta_k)=1,\qquad R(k,\delta_0)=0. $$ Thus $R(k,\delta_k)\le R(k,\delta_0)+\varepsilon$ would require $1\le\varepsilon$. For every $0\le\varepsilon<1$, this replacement inequality fails. The class $C$ is therefore not epsilon-complete for such an $\varepsilon$, despite the pointwise convergence of its sequence of risk functions to the risk of $\delta_0$.

In the full decision problem, $\delta_0$ itself is available and has risk zero, so it dominates every rule: all risks are nonnegative. The failure is specifically in trying to use $C$ as the replacement class. Pointwise risk closure does not ensure that a suitable replacement is actually a member of that class.

There is a useful pointwise statement, but it has a different conclusion. Suppose $R(\theta,\delta_m)\to r(\theta)$ for every $\theta$, and suppose that for every $m$ and every $\theta$, $$ R(\theta,\delta_m)\le R(\theta,\delta)+\varepsilon. $$ Taking the limit in this numerical inequality at each fixed $\theta$ gives $r(\theta)\le R(\theta,\delta)+\varepsilon$. This argument needs no uniform convergence: the inequality already holds for each parameter and each index. But to call $r$ the risk of a replacement rule, one must also have a rule in the proposed class whose risk is $r$. A limit function is not automatically an attainable risk function.

Key distinction: If a risk limit is attainable, pointwise limits can preserve inequalities that already hold at every parameter. But pointwise convergence alone does not establish a uniform approximation bound, does not produce an attaining rule, and does not guarantee an epsilon-complete replacement class.

Common Mistake: Letting the Parameter Depend on the Index

A tempting but incorrect argument is: “For every fixed $\theta$, the risks eventually agree, so the rules are uniformly close.” The premise concerns a fixed $\theta$; uniform closeness requires one error bound to hold for all $\theta$ at the same index. In the example, choosing $\theta=m$ for the $m$th rule gives an error of 1, so the supremum error never falls.

A related mistake is to infer that a pointwise risk limit is itself an available rule in a candidate class. Risk functions can have limits that the class does not contain. Before using a limit as a replacement, verify both the required risk inequality and the existence of a rule in the class with the proposed risk.

Identify the replacement claim.
Check whether it requires a risk inequality at each parameter or a uniform additive error over all parameters.

Check the type of convergence.
Pointwise convergence controls each fixed parameter; uniform convergence controls the supremum error.

Check attainability.
Even if a limiting risk function has the desired property, verify that a rule in the proposed class actually has that risk.

Check Your Understanding

Use the moving-spike example and the definitions above to answer the following.

For a fixed positive integer $k$, what is $R(k,\delta_m)$ when $m=k$, and what is it when $m\ne k$?
Why does $R(\theta,\delta_m)\to R(\theta,\delta_0)$ for every fixed real $\theta$, even though the uniform error is always 1?
What happens to the worst-case risk of $\delta_m$ as $m$ increases? Compare it with the worst-case risk of $\delta_0$.
For the weighted distance in the second example, why does the distance tend to zero while the supremum distance does not?
State one additional fact, beyond pointwise convergence of risks, that is needed before treating a limiting risk function as a replacement from a particular class.