Weak Convergence of Priors and Bayes Risks

Why Weak Convergence Does Not Immediately Settle Bayes Risk

A prior assigns probability to parameter values, and a rule has a risk at each such value. Weak convergence is useful because it controls integrals of bounded continuous functions: for any fixed rule with bounded continuous risk, its prior-averaged risk converges. Bayes risk, however, is an infimum over rules of those integrals. Convergence for each fixed rule does not, by itself, justify exchanging a limit and an infimum.

The distinction matters when a sequence of priors is chosen to make Bayes risks approach their supremum. If Bayes risk is continuous along that sequence, then a weak limit preserves the limiting value and can attain the supremum. The key is to ensure enough uniformity across the rules being optimized.

Definition: Let $\Theta$ be a metric parameter space, let $\mathcal D$ be a collection of decision rules, and let $R(\theta,\delta)$ be the risk of rule $\delta$ at parameter $\theta$. For a probability measure $\pi$ on $\Theta$, the Bayes risk is $$ r(\pi)=\inf_{\delta\in\mathcal D}\int_\Theta R(\theta,\delta)\,\pi(d\theta). $$ A sequence of priors $\pi_n$ converges weakly to $\pi$, written $\pi_n\Rightarrow\pi$, if $\int g\,d\pi_n\to\int g\,d\pi$ for every bounded continuous function $g:\Theta\to\mathbb R$.

The integral $\int R(\theta,\delta)\,\pi(d\theta)$ is the prior-averaged risk of a particular rule. Weak convergence gives convergence of this quantity for each fixed $\delta$ whenever its risk is bounded and continuous. To control the infimum over $\mathcal D$, we need convergence that is uniform across the whole risk family $\{R(\cdot,\delta):\delta\in\mathcal D\}$.

A Sufficient Condition for Bayes-Risk Convergence

Here is a useful condition that makes the needed uniformity explicit. Suppose the risks are bounded by a common finite constant and change uniformly little when the parameter changes by a small amount. Also suppose the priors do not place substantial mass arbitrarily far out in the parameter space.

Theorem: Let $\Theta$ be a metric space, and let $\pi_n\Rightarrow\pi$. Suppose there is a finite $M$ such that $|R(\theta,\delta)|\le M$ for every $\theta\in\Theta$ and $\delta\in\mathcal D$. Suppose every $R(\cdot,\delta)$ is continuous and the family of risks is equicontinuous: for every $\varepsilon>0$, there is a $\eta>0$ such that $$ d(\theta,\theta')<\eta \quad\Longrightarrow\quad |R(\theta,\delta)-R(\theta',\delta)|<\varepsilon $$ for every $\delta\in\mathcal D$. Finally, suppose the sequence of priors and its limit are uniformly tight: for every $\tau>0$, there is a compact $K\subseteq\Theta$ such that $\sup_n\pi_n(K^c)\le\tau$ and $\pi(K^c)\le\tau$. Then $$ r(\pi_n)\longrightarrow r(\pi). $$

The uniform tightness condition is automatic when $\Theta$ is compact. For probability measures on a Polish space, weak convergence also implies uniform tightness by Prokhorov’s theorem. The proof below uses tightness to control the risks outside a compact set, where equicontinuity alone would not provide a finite approximation.

Approximate the risk family on a compact set.
Uniform tightness gives a compact set carrying all but a small amount of each prior’s mass. On that compact set, bounded equicontinuous functions have a finite approximation in the uniform norm.

Use weak convergence for finitely many risks.
Each representative risk is bounded and continuous, so its integral converges. Since there are finitely many representatives, the largest of their integral differences also converges to zero.

Pass from integrated risks to their infima.
Uniform convergence of the integrated risks over all rules bounds the difference between their infima.

Proof. Write $A_n(\delta)=\int R(\theta,\delta)\,\pi_n(d\theta)$ and $A(\delta)=\int R(\theta,\delta)\,\pi(d\theta)$. We first show that $$ \sup_{\delta\in\mathcal D}|A_n(\delta)-A(\delta)|\longrightarrow 0. $$ Fix a positive error tolerance. By uniform tightness, choose a compact $K$ such that both $\pi_n(K^c)\le\tau$ for every $n$ and $\pi(K^c)\le\tau$, where $\tau>0$ can be chosen as small as desired. By boundedness and equicontinuity, the restrictions of the risks to $K$ form a totally bounded family in the uniform norm, by the Arzelà–Ascoli theorem. Thus, for any $\gamma>0$, there are finitely many rules $\delta_1,\ldots,\delta_m$ such that for every $\delta\in\mathcal D$, some $j$ satisfies $$ \sup_{\theta\in K}|R(\theta,\delta)-R(\theta,\delta_j)|\le\gamma. $$ For this $\delta_j$, split each integral over $K$ and $K^c$. The bound $|R|\le M$ gives $$ \int |R(\theta,\delta)-R(\theta,\delta_j)|\,\pi_n(d\theta) \le \gamma+2M\tau, $$ and the same bound holds with $\pi$ in place of $\pi_n$. Consequently, $$ |A_n(\delta)-A(\delta)| \le |A_n(\delta_j)-A(\delta_j)|+2\gamma+4M\tau. $$ For each of the finitely many representatives, weak convergence gives $A_n(\delta_j)\to A(\delta_j)$. Hence their maximum absolute difference tends to zero. Since $\gamma$ and $\tau$ can be made arbitrarily small, the asserted uniform convergence follows.

For any two bounded real families $\{a_\delta\}$ and $\{b_\delta\}$, $$ \left|\inf_{\delta}a_\delta-\inf_{\delta}b_\delta\right| \le\sup_{\delta}|a_\delta-b_\delta|. $$ Indeed, if the right side is $c$, then $b_\delta-c\le a_\delta\le b_\delta+c$ for every $\delta$; taking infima gives the inequality. Apply it to $A_n$ and $A$. The uniform convergence just proved implies $|r(\pi_n)-r(\pi)|\to0$. $\square$

Worked Examples

Worked Example: Bayes Risk for Squared-Error Loss

There is no observation, the parameter space is $\Theta=[0,1]$, and a rule chooses an action $a\in[0,1]$. The loss is $R(\theta,a)=(a-\theta)^2$. Find the Bayes risk for a prior $\pi$, and explain why weak convergence of priors preserves it.

Let $\mu=E_\pi[\theta]$. For any action $a$, $$ \int (a-\theta)^2\,\pi(d\theta) =a^2-2a\mu+E_\pi[\theta^2] =(a-\mu)^2+E_\pi[\theta^2]-\mu^2. $$ Because $\mu\in[0,1]$, the minimizing action $a=\mu$ is allowed. Therefore $$ r(\pi)=E_\pi[\theta^2]-(E_\pi[\theta])^2=\operatorname{Var}_\pi(\theta). $$ The risks are bounded by $1$. They are also equicontinuous in $\theta$, since for every $a\in[0,1]$, $$ |(a-\theta)^2-(a-\theta')^2| =|\theta-\theta'|\,|2a-\theta-\theta'| \le 2|\theta-\theta'|. $$ The parameter space is compact, so the theorem applies to any weakly convergent sequence of priors. For instance, $\pi_n=\delta_{1/n}\Rightarrow\delta_0$, and $r(\pi_n)=0\to0=r(\delta_0)$.

Worked Example: A Limiting Prior Attains the Supremum

In the same squared-error problem, show that the supremum of Bayes risk over all priors on $[0,1]$ is attained. Then exhibit a sequence of priors whose Bayes risks converge to that supremum.

For any prior, write $\mu=E_\pi[\theta]$. Since $0\le\theta\le1$, we have $\theta^2\le\theta$, and hence $$ r(\pi)=E_\pi[\theta^2]-\mu^2 \le \mu-\mu^2 =\frac14-\left(\mu-\frac12\right)^2 \le\frac14. $$ The prior $\pi_*=\tfrac12\delta_0+\tfrac12\delta_1$ has mean $1/2$ and second moment $1/2$, so $$ r(\pi_*)=\frac12-\left(\frac12\right)^2=\frac14. $$ Thus the supremum is $1/4$, attained by $\pi_*$.

To see the limiting-prior argument directly, set $\pi_n=(1-1/n)\pi_*+(1/n)\delta_{1/3}$. For any bounded continuous $g$, $\int g\,d\pi_n\to\int g\,d\pi_*$, so $\pi_n\Rightarrow\pi_*$. Its first and second moments are $$ E_{\pi_n}[\theta]=\frac12-\frac{1}{6n}, \qquad E_{\pi_n}[\theta^2]=\frac12-\frac{7}{18n}. $$ Therefore $$ r(\pi_n)=\frac14-\frac{2}{9n}-\frac{1}{36n^2} \longrightarrow\frac14=r(\pi_*). $$ The weak limit of this sequence of priors attains the supremum of the Bayes risks.

Why Individual Continuity Is Not Enough

The equicontinuity condition in the theorem cannot simply be replaced by the statement that each risk is bounded and continuous. With infinitely many rules, the rule that nearly minimizes integrated risk can change with the prior. Pointwise convergence of integrated risks does not control such a changing sequence of rules.

Worked Example: Continuous Risks but Discontinuous Bayes Risk

Let $\Theta=[0,1]$, with no observation, and let the available actions be the positive integers $k=1,2,\ldots$. Define the loss by $R(\theta,k)=e^{-k\theta}$. Show that weak convergence of priors need not preserve Bayes risk in this problem.

For each fixed $k$, the risk $e^{-k\theta}$ is continuous in $\theta$ and lies between $0$ and $1$. Consider $\pi_n=\delta_{1/n}$, which converges weakly to $\pi=\delta_0$. Under $\pi_n$, the integrated risk for action $k$ is $e^{-k/n}$, so $$ r(\pi_n)=\inf_{k\ge1}e^{-k/n}=0. $$ The infimum is zero even though no finite $k$ makes the risk zero. Under $\pi=\delta_0$, every action has risk $e^0=1$, and therefore $r(\pi)=1$. Thus $r(\pi_n)\not\to r(\pi)$.

The family is not equicontinuous: for $\theta=0$ and $\theta'=1/k$, the distance is $1/k\to0$, but the risk difference for action $k$ is $1-e^{-1}$, which does not tend to zero. This is exactly the uniformity failure that allows the Bayes risk to jump.

When a Supremum of Bayes Risks Is Attained

Let $\mathcal P$ be a collection of priors and suppose $S=\sup_{\pi\in\mathcal P}r(\pi)$ is finite. A sequence $\pi_n\in\mathcal P$ with $r(\pi_n)\to S$ is called a maximizing sequence. If a subsequence $\pi_{n_j}$ converges weakly to a prior $\pi_*$, and the conditions of the theorem hold along that subsequence, then $$ r(\pi_*)=\lim_{j\to\infty}r(\pi_{n_j})=S. $$ If $\pi_*\in\mathcal P$, it attains the supremum over $\mathcal P$. Such a prior is often called least favorable when the supremum is taken over the priors in the decision problem.

This argument separates two questions. Bayes-risk continuity shows that a weak limit preserves the value of a maximizing sequence. It does not, on its own, guarantee that a maximizing sequence has a weakly convergent subsequence. Compactness of the prior family is one sufficient way to ensure such a subsequence exists; without an appropriate compactness condition, a maximizing sequence may have no limit within the space of priors under consideration.

Common Mistake

Incorrect reasoning: “For every fixed rule, weak convergence makes its integrated risk converge, so the infimum over all rules must converge too.”
Correction: Pointwise convergence over rules does not generally commute with taking an infimum. Establish uniform convergence over the rule class, for example with bounded equicontinuous risks and uniformly tight priors, before concluding that Bayes risks converge.

Key takeaway: Weak convergence preserves the integrated risk of each fixed bounded continuous risk function. To conclude convergence of Bayes risks, control the whole risk family uniformly. Once that continuity is established, any weak limit of a maximizing sequence preserves its limiting Bayes-risk value.

Check Your Understanding

Try these questions before moving on.

State the definition of weak convergence of priors used in this tutorial.
Why does convergence of integrated risk for each fixed rule not automatically imply convergence of Bayes risk?
In the theorem, what roles do equicontinuity and uniform tightness play in the proof?
For squared-error loss on $[0,1]$, why does the prior concentrated equally at $0$ and $1$ have Bayes risk $1/4$?
In the exponential-loss example, identify the precise failure that permits the Bayes risk to be discontinuous.
What additional fact about a maximizing sequence is needed, beyond Bayes-risk continuity, to obtain a limiting prior that attains the supremum?