Why Pointwise Risk Convergence Is Not Enough

Why the Worst Case Can Move

A risk function records the expected loss of a procedure at each parameter value. Pointwise convergence says that, at every fixed parameter, the risks eventually become close to their limiting values. A worst-case risk asks a different question: how large can the risk be over all parameter values? The parameter giving the largest risk may change with the approximation index, so pointwise control need not control the worst case.

This distinction matters when assessing sequences of procedures and when comparing minimax values of approximating risk criteria. A small risk at each fixed parameter does not rule out a large risk at a parameter that keeps moving as the approximation changes.

Definition: Let $\Theta$ be a parameter space and let $R_n:\Theta\to\mathbb{R}$ be a sequence of risk functions. The sequence converges pointwise to $R$ if, for every fixed $\theta\in\Theta$, $R_n(\theta)\to R(\theta)$. The worst-case risk of $R_n$ is $\sup_{\theta\in\Theta}R_n(\theta)$, when this supremum is defined. Pointwise convergence alone does not assert convergence of these suprema.

The quantifiers explain the gap. Pointwise convergence allows the index after which the risks are close to depend on $\theta$. To control the worst case by the same argument at every parameter, one would need a bound that works uniformly over $\Theta$. The following examples make the distinction concrete.

A Moving Spike on a Countable Parameter Space

Worked Example: A Procedure That Misses One Parameter at a Time

Let $\Theta=\{1,2,3,\ldots\}$. The observation is uninformative: it always equals 0. The action space is also $\Theta$, and the loss is $L(\theta,a)=1$ if $a=\theta$, and 0 otherwise. For each $n$, use the procedure $\delta_n$ that always chooses action $n$.

Because the procedure is deterministic and the observation contains no information, its risk at $\theta$ is just its loss there:

R(\theta,\delta_n)= \begin{cases} 1,&\theta=n,\\ 0,&\theta\ne n. \end{cases}

Fix any $\theta$. For every $n>\theta$, the action $n$ differs from $\theta$, so $R(\theta,\delta_n)=0$. Thus $R(\theta,\delta_n)\to 0$ for every fixed $\theta$. But for each $n$, the risk at the moving parameter $\theta_n=n$ is 1, and all risks are at most 1. Hence

\sup_{\theta\in\Theta}R(\theta,\delta_n)=1 \quad\text{for every }n, \qquad \sup_{\theta\in\Theta}0=0.

The pointwise limiting function is identically zero, while the worst-case risks remain equal to 1. In this decision problem, the failure is not an artifact of inventing arbitrary functions: each $R(\theta,\delta_n)$ is the risk of an actual procedure under a fixed loss.

What Pointwise Convergence Does Guarantee

Although pointwise convergence does not give convergence of worst-case risks, it does give a one-sided bound. This bound helps diagnose what has gone wrong: the limiting worst-case risk cannot exceed the lower limit of the approximating worst-case risks.

Proposition: Suppose $R_n(\theta)\to R(\theta)$ for every $\theta\in\Theta$, and suppose $M_n=\sup_{\theta\in\Theta}R_n(\theta)$ is finite for each $n$. Then $$ \sup_{\theta\in\Theta}R(\theta)\leq \liminf_{n\to\infty}M_n. $$ This statement allows the left side or the lower limit to be infinite.

Proof. Fix $\theta\in\Theta$. Since $R_n(\theta)\leq M_n$ for every $n$, taking the lower limit gives

R(\theta)=\liminf_{n\to\infty}R_n(\theta) \leq \liminf_{n\to\infty}M_n.

The right side does not depend on $\theta$. Taking the supremum over $\theta$ proves the claim. $\square$

This proposition supplies no matching upper bound. In the moving-spike example, it says $0\leq 1$, which is true but does not force the worst-case risks toward 0. An upper bound needs additional control that prevents large risks from escaping to changing parameter values.

Proposition: If $\Theta$ is finite and $R_n(\theta)\to R(\theta)$ for every $\theta\in\Theta$, then $$ \sup_{\theta\in\Theta}R_n(\theta)\longrightarrow \sup_{\theta\in\Theta}R(\theta). $$

Proof. Write $\Theta=\{\theta_1,\ldots,\theta_m\}$. For any two real vectors $x=(x_1,\ldots,x_m)$ and $y=(y_1,\ldots,y_m)$,

\left|\max_{1\leq i\leq m}x_i-\max_{1\leq i\leq m}y_i\right| \leq \max_{1\leq i\leq m}|x_i-y_i|.

Indeed, $x_i\leq y_i+\max_j|x_j-y_j|$ for each $i$, so $\max_i x_i\leq\max_i y_i+\max_j|x_j-y_j|$; exchanging $x$ and $y$ gives the reverse bound. Apply this inequality with $x_i=R_n(\theta_i)$ and $y_i=R(\theta_i)$. Since there are finitely many coordinates and each converges, their maximum absolute difference tends to zero. The displayed convergence follows. $\square$

Finiteness prevents a worst-case parameter from continually escaping beyond the fixed list of parameters. On an infinite parameter space, pointwise convergence supplies no such protection.

A Continuous Moving Spike

Worked Example: Continuous Risks on an Interval

Let $\Theta=[0,1]$, and for each positive integer $n$ define

R_n(\theta)=\max\{1-2n|\theta-1/n|,\,0\}.

This is a continuous, nonnegative risk function. It is positive only when $|\theta-1/n|<1/(2n)$, so its support is contained in the interval $[1/(2n),3/(2n)]$, intersected with $[0,1]$. At $\theta=1/n$, its value is 1, and everywhere its value is at most 1. Therefore

\sup_{\theta\in[0,1]}R_n(\theta)=R_n(1/n)=1.

Now fix $\theta\in[0,1]$. At $\theta=0$, $R_n(0)=0$ for every $n$. If $\theta>0$, then $3/(2n)<\theta$ for all sufficiently large $n$, so $\theta$ lies outside the support and $R_n(\theta)=0$. Thus $R_n(\theta)\to0$ at every fixed $\theta$, even though the supremum remains 1.

The parameter space is compact, and every risk function is continuous. Those features do not prevent this failure: the spikes get narrower as they move toward 0. Pointwise convergence does not limit how narrow or how rapidly moving such high-risk regions may be.

Why Minimax Values Can Also Fail to Converge

For a risk surface $R_n(\theta,a)$, indexed by parameter $\theta$ and available procedure or action $a$, its minimax value is

V_n=\inf_{a\in\mathcal{A}}\sup_{\theta\in\Theta}R_n(\theta,a).

The infimum over procedures does not remove the issue with a supremum over parameters. The next example gives pointwise convergence of every risk at every fixed parameter-action pair, while the minimax values do not converge to the minimax value of the limiting risk surface.

Worked Example: Pointwise-Convergent Risk Surfaces, Different Minimax Values

Let $\Theta=\{1,2,3,\ldots\}$, and let the action space be $\mathcal{A}=\{a_0,a_1\}$. For each $n$, define the risk criterion

R_n(\theta,a)= \begin{cases} 1,&\theta=n,\\ 0,&\theta\ne n,\ a=a_0,\\ 1/2,&\theta\ne n,\ a=a_1. \end{cases}

At any fixed $\theta$, once $n>\theta$, the exceptional case $\theta=n$ no longer applies. Therefore the pointwise limiting surface is $R(\theta,a_0)=0$ and $R(\theta,a_1)=1/2$. Its minimax value is

V=\inf_{a\in\{a_0,a_1\}}\sup_{\theta\in\Theta}R(\theta,a) =\min\{0,1/2\}=0.

For each $n$, however, both actions have risk 1 at $\theta=n$. Their worst-case risks are therefore both 1, and

V_n=\inf_{a\in\{a_0,a_1\}}\sup_{\theta\in\Theta}R_n(\theta,a) =\min\{1,1\}=1.

Consequently, $V_n=1$ for every $n$, while $V=0$. The high risk occurs at a different parameter for each criterion, and no choice between the two actions avoids it. These are valid nonnegative, bounded risk criteria; they can, for example, be represented by a decision problem with an uninformative observation and the displayed criteria as losses. The example shows that pointwise convergence of risk surfaces, by itself, does not justify passing to a minimax value.

Common Mistake: Treating “Every Fixed Parameter” as “All Parameters at Once”

A tempting but invalid argument is: “For each $\theta$, $R_n(\theta)$ becomes small, so eventually the risk is small for every $\theta$; hence the supremum becomes small.” The conclusion changes the quantifiers. Pointwise convergence says that for each $\theta$ and each tolerance, there is an index after which the risk at that $\theta$ is small. It does not say there is one index that works for all $\theta$. In the first example, for each fixed $\theta$, the risk is zero whenever $n>\theta$, but at every index $n$ the risk is still 1 at $\theta=n$.

When checking an argument about worst-case risk or minimax values, track the parameter that nearly maximizes the risk. If that parameter is allowed to depend on $n$, pointwise convergence at fixed parameters may not constrain it. The issue is not whether the risk is small at each fixed location; it is whether high-risk locations can keep appearing elsewhere.

Key takeaway: Pointwise risk convergence controls each fixed parameter separately. It does not, without further assumptions, control the worst-case risk or the minimax value. Always check whether a worst-case parameter can move with the approximation index.

Check Your Understanding

Use the distinction between fixed parameters and moving worst-case parameters to answer these questions.

For $R_n(\theta)=1\{\theta=n\}$ on the positive integers, what is the pointwise limit, and what is $\sup_\theta R_n(\theta)$ for each $n$?
Why does the finite-parameter proposition rule out the same pointwise-convergence failure on a fixed finite parameter space?
In the continuous-spike example, identify a parameter $\theta_n$ attaining the worst-case risk at index $n$, and verify its risk.
In the two-action minimax example, compute the worst-case risk of each action both before and after taking the pointwise limit.
State the quantifier difference between pointwise convergence and a bound that holds uniformly over all parameter values.