Discontinuous Losses and Failed Risk Limits

Why a Weak Limit Can Change Risk

In complete-class arguments, a sequence of decision rules may have a weakly convergent subsequence, and one would like the limiting rule to retain a risk bound satisfied by every rule in the sequence. Weak convergence controls integrals of bounded continuous functions, but a loss function need not be continuous. At a discontinuity, the risk of the limit can therefore be larger than the limiting risks.

The distinction is not about whether the action space is compact or whether the loss is bounded. The example below has a compact action space and a bounded, measurable loss. The failure comes from the loss assigning a special value to an action that nearby rules approach.

Decision Rules and Weak Convergence

For the examples, take a decision problem with one parameter value, no informative data, and action space $[0,1]$. A randomized decision rule is then a probability distribution $Q$ on the action space. Its risk is the expected loss under that distribution.

Definition: A sequence of probability distributions $Q_n$ on $[0,1]$ converges weakly to $Q$, written $Q_n\Rightarrow Q$, if $$ \int g(a)\,Q_n(da)\longrightarrow \int g(a)\,Q(da) $$ for every bounded continuous function $g:[0,1]\to\mathbb{R}$. For a measurable loss $\ell$, the risk of rule $Q$ is $R(Q)=\int \ell(a)\,Q(da)$, when this integral is defined.

The definition tests integrals against continuous functions, not against every bounded measurable function. Thus weak convergence alone gives no guarantee that $R(Q_n)\to R(Q)$ for a discontinuous loss. In particular, it need not preserve an inequality of the form $R(Q_n)\le r$.

Worked Example: A Risk Bound Lost at a Single Action

Let the loss be $\ell(a)=1$ when $a=c$, and $\ell(a)=0$ otherwise, where $c=1/2$. For each integer $n\ge1$, use the deterministic rule $Q_n=\delta_{c+1/(2n)}$, where $\delta_x$ denotes a point mass at $x$. Every support point is a valid action: $1/2<c+1/(2n)\le1$.

To check weak convergence, take any bounded continuous $g$. Then $$ \int g(a)\,Q_n(da)=g\left(c+\frac{1}{2n}\right)\longrightarrow g(c)=\int g(a)\,\delta_c(da). $$ Hence $Q_n\Rightarrow\delta_c$.

For every $n$, the selected action differs from $c$, so $$ R(Q_n)=\ell\left(c+\frac{1}{2n}\right)=0. $$ The limiting rule selects $c$, and therefore $R(\delta_c)=\ell(c)=1$. For example, the reference rule $Q^\star=\delta_0$ has risk $R(Q^\star)=0$. Every $Q_n$ satisfies $R(Q_n)\le R(Q^\star)$, but the weak limit does not: $R(\delta_c)=1>0=R(Q^\star)$. Thus even an exact risk-domination inequality at every stage need not pass to the weak limit.

Randomization Does Not Remove the Problem

The same issue appears when rules randomize. Weak convergence may place probability on a discontinuity in the limit even when the sequence assigns little or no probability to that action. The risk depends on the loss at the action itself, not just on how close the actions are.

Worked Example: Probability Accumulating at the Discontinuity

Keep $\ell(a)=\mathbf{1}_{\{c\}}(a)$ with $c=1/2$, and for $n\ge2$ define $$ Q_n=\left(1-\frac{1}{n}\right)\delta_{c+1/(2n)}+\frac{1}{n}\delta_c. $$ Both support points lie in $[0,1]$. The mass at $c$ is $1/n$, so direct calculation gives $R(Q_n)=1/n$.

For any bounded continuous $g$, $$ \int g\,dQ_n=\left(1-\frac{1}{n}\right)g\left(c+\frac{1}{2n}\right)+\frac{1}{n}g(c)\longrightarrow g(c). $$ Thus $Q_n\Rightarrow\delta_c$, whose risk is $1$. The risks $1/n$ tend to zero, while the limit risk is one. The vanishing mass at the discontinuity does not prevent the limit distribution from concentrating there.

This example also shows why inspecting only the probability assigned to the exceptional action at finite $n$ is not enough. Weak convergence concerns the distribution as a whole; the limiting distribution can have an atom at a point that carried only a vanishing fraction of the mass along the sequence.

Lower Semicontinuity Supplies the Needed Inequality

A standard condition that rules out this upward jump in risk is lower semicontinuity of the loss. For a function $\ell$, lower semicontinuity means that $\ell(a)\le\liminf_{m\to\infty}\ell(a_m)$ whenever $a_m\to a$. It permits the value at a point to be lower than nearby values, but not higher in a way that creates a downward jump nearby and a larger value at the limit.

Theorem: Let $K$ be a compact metric space, let $Q_n\Rightarrow Q$ be probability distributions on $K$, and let $\ell:K\to[0,M]$ be lower semicontinuous for some finite $M$. Then $$ \int \ell\,dQ\le \liminf_{n\to\infty}\int \ell\,dQ_n. $$ Consequently, if $\int \ell\,dQ_n\le r$ for every $n$, then $\int \ell\,dQ\le r$.

Here is a proof for this bounded-loss setting. For each positive integer $k$, define $$ \ell_k(a)=\inf_{b\in K}\{\ell(b)+k\,d(a,b)\}, $$ where $d$ is the metric on $K$. Each $\ell_k$ is bounded and continuous: it is nonnegative, at most $M$ by taking $b=a$, and is $k$-Lipschitz by the triangle inequality. Also, $\ell_k(a)\le\ell(a)$, and $\ell_k(a)$ is nondecreasing in $k$.

For each fixed $a$, $\ell_k(a)\to\ell(a)$. To see this, suppose instead that for some $\varepsilon>0$ and arbitrarily large $k$, $\ell_k(a)\le\ell(a)-\varepsilon$. By the definition of infimum, choose $b_k$ with $$ \ell(b_k)+k\,d(a,b_k)<\ell(a)-\frac{\varepsilon}{2}. $$ Since $\ell(b_k)\ge0$, this implies $d(a,b_k)<\ell(a)/k$, so $b_k\to a$. The same inequality gives $\ell(b_k)<\ell(a)-\varepsilon/2$, contradicting lower semicontinuity at $a$. Thus the claimed pointwise convergence holds.

For a fixed $k$, weak convergence applies to the bounded continuous function $\ell_k$. Since $\ell_k\le\ell$, $$ \int\ell_k\,dQ=\lim_{n\to\infty}\int\ell_k\,dQ_n \le\liminf_{n\to\infty}\int\ell\,dQ_n. $$ Now let $k\to\infty$. The functions $\ell_k$ increase pointwise to $\ell$, so the monotone convergence theorem gives $\int\ell_k\,dQ\to\int\ell\,dQ$. The displayed inequality therefore proves the theorem. If every risk in the sequence is at most $r$, its limit inferior is at most $r$, yielding the stated risk bound for $Q$. $\square$

Worked Example: Checking the Lower-Semicontinuous Case

On $[0,1]$, define $\widetilde{\ell}(a)=0$ at $c=1/2$ and $\widetilde{\ell}(a)=1$ at every other action. This loss is lower semicontinuous: for $0\le t<1$, the sublevel set $\{a:\widetilde{\ell}(a)\le t\}$ is the closed set $\{c\}$; for $t<0$ it is empty, and for $t\ge1$ it is all of $[0,1]$.

Use $Q_n=\delta_{c+1/(2n)}$ for $n\ge1$, as before. These rules converge weakly to $\delta_c$. For each $n$, the action is not $c$, so $\int\widetilde{\ell}\,dQ_n=1$; at the limit, $\int\widetilde{\ell}\,d\delta_c=0$. In particular, the limit risk $0$ is no greater than the limit inferior of the sequence risks, which is $1$, as the theorem requires.

The contrast is important. The first loss assigned the exceptional point a higher value than its nearby values and was not lower semicontinuous. This loss assigns it a lower value, is lower semicontinuous, and satisfies the risk inequality even though the risks do not converge.

Applying the Result to Statistical Rules

In a statistical decision problem, a rule is generally a probability kernel from the observation space to the action space, and risk averages the loss over both the data and the chosen action. A limit theorem for action distributions applies directly when the relevant induced distributions of actions (or of data and actions jointly) converge weakly and the loss is lower semicontinuous on the space carrying those joint distributions. Under those conditions, the theorem gives a limit-inferior risk inequality at each fixed parameter value.

In particular, if for a fixed $\theta$ the sequence of rules has risks no greater than a reference risk, and the hypotheses of the lower-semicontinuity result hold for the corresponding distributions and loss, then the limiting rule has risk no greater than that reference risk at that $\theta$. The inequality follows from the limit-inferior bound, not from an assumption that the risks themselves converge. When the loss is discontinuous without lower semicontinuity, the point-mass example shows that this inference can fail.

Common Mistake

Wrong reasoning: “The rules converge weakly, and every rule has risk at most the reference risk, so the limit has risk at most the reference risk.” Weak convergence only guarantees convergence of integrals for bounded continuous functions; a discontinuous loss is not covered by that guarantee.
Correction: Check the loss and the relevant weak convergence hypotheses. A bounded lower-semicontinuous loss gives the limit-inferior inequality proved above. Without that condition, test whether the loss assigns a larger value at an action where probability can accumulate in the limit.
Another error: Treating compactness of the action space or boundedness of the loss as a substitute for lower semicontinuity. In the counterexample, both compactness and boundedness hold, yet the risk bound fails.

Key takeaway: Weak convergence does not by itself preserve risk inequalities. In the compact, bounded-loss setting, lower semicontinuity provides the crucial inequality $R(Q)\le\liminf_n R(Q_n)$; a discontinuity that violates it can destroy risk domination at the limit.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

For the loss $\ell(a)=\mathbf{1}_{\{1/2\}}(a)$, why does $\delta_{1/2+1/(2n)}$ converge weakly to $\delta_{1/2}$, even though its risk does not converge to the limiting risk?
In the randomized example, calculate the risk of $Q_n$ and the risk of its weak limit. Which support point creates the nonzero risk at finite $n$?
Why is the loss equal to zero at $1/2$ and one elsewhere lower semicontinuous, while the reverse assignment is not?
State the risk inequality that lower semicontinuity gives when $Q_n\Rightarrow Q$. Does it require the risks to converge?
Why are compactness of the action space and boundedness of the loss alone insufficient to pass risk domination to a weak limit?