Approximating Bayes Rules by Finite Problems

Why Approximate a Bayes Problem by Finite Problems?

A Bayes rule minimizes expected loss under a prior distribution. In many problems, the parameter space and the available actions are continua, so the minimization is not a finite calculation. Finite approximations replace a prior by one supported on finitely many parameter values and restrict actions to a finite grid. They can make the optimization explicit, but their value depends on whether their Bayes risks and rules genuinely approximate those of the original problem.

The key distinction is between approximating the problem and merely solving a convenient finite problem. Convergence requires control of how risks change with the parameter and action, as well as enough compactness to obtain a limiting rule. We develop a risk-based result that makes these requirements precise.

Set-Up and Finite Approximations

Definition: Let \(\Theta\) be a parameter space, \(\mathcal{D}\) a set of decision rules, and \(R(\theta,\delta)\) the risk of rule \(\delta\in\mathcal{D}\) at parameter \(\theta\). For a prior probability measure \(\pi\), the integrated risk is \(r(\pi,\delta)=\int_\Theta R(\theta,\delta)\,\pi(d\theta)\). A Bayes rule for \(\pi\) is a rule minimizing this integrated risk over \(\mathcal{D}\). Its minimum value is the Bayes risk.

A finite parameter approximation is a probability measure \(\pi_m\) supported on a finite set \(\Theta_m\subset\Theta\). The sequence \(\pi_m\) approximates \(\pi\) if it converges weakly to \(\pi\): for every bounded continuous function \(g\) on \(\Theta\), \(\int g\,d\pi_m\to\int g\,d\pi\). A finite action approximation restricts decision rules to a set \(\mathcal{D}_m\) whose actions lie in a finite subset \(\mathcal{A}_m\) of the action space.

For example, if a rule is a probability kernel from observations to a compact action space, a finite action grid can be used by mapping each action to a nearby grid point. This preserves the rule's dependence on the observation while changing its possible actions. For bounded loss uniformly continuous in the action, the resulting change in risk is uniformly small as the grid spacing shrinks.

Worked Example: A Uniform Prior and Squared-Error Loss

Suppose there is no observation, \(\Theta=[0,1]\), the action is \(a\in[0,1]\), the loss is \(L(\theta,a)=(a-\theta)^2\), and the prior is uniform on \([0,1]\). The risk at \(\theta\) is \(R(\theta,a)=(a-\theta)^2\). The integrated risk is

r(\pi,a)=\int_0^1(a-\theta)^2\,d\theta =a^2-a+\frac{1}{3} =\left(a-\frac{1}{2}\right)^2+\frac{1}{12}.

Thus the original Bayes action is \(a=1/2\), with Bayes risk \(1/12\). Now take \(m\) equally weighted midpoint parameter values \(\theta_j=(j-\tfrac12)/m\), for \(j=1,\ldots,m\), and let \(m\) be even so the finite action grid includes \(1/2\). The finite-prior integrated risk is minimized at the mean of the grid points:

\bar{\theta}_m=\frac{1}{m}\sum_{j=1}^m\frac{j-\tfrac12}{m}=\frac12.

The finite-problem Bayes action is therefore \(1/2\). Its finite-prior Bayes risk is the variance of the grid points, \(\frac{1}{m}\sum_{j=1}^m(\theta_j-\tfrac12)^2=\frac{m^2-1}{12m^2}\), which tends to \(1/12\). Here both the finite Bayes action and its Bayes risk recover the original answer as the parameter grid is refined.

Uniform Control of Integrated Risk

Weak convergence of priors controls integrals of each fixed bounded continuous function. To compare Bayes rules, however, the relevant function \(R(\cdot,\delta)\) changes with \(\delta\). The following lemma supplies the needed uniform control.

Lemma: Let \(\Theta\) be compact metric, let \(\pi_m\) converge weakly to \(\pi\), and suppose the family of functions \(\{R(\cdot,\delta):\delta\in\mathcal{D}\}\) is bounded and equicontinuous on \(\Theta\). Then \(\sup_{\delta\in\mathcal{D}}|r(\pi_m,\delta)-r(\pi,\delta)|\to0\).

Proof. Write \(f_\delta(\theta)=R(\theta,\delta)\). Equicontinuity means that for every \(\varepsilon>0\), there is a distance \(s>0\) such that \(|f_\delta(\theta)-f_\delta(\theta')|<\varepsilon\) whenever the distance between \(\theta\) and \(\theta'\) is less than \(s\), for every \(\delta\). Boundedness and equicontinuity on compact \(\Theta\) imply that this family has a finite \(\varepsilon\)-net in the uniform norm: choose a finite sufficiently fine cover of \(\Theta\), and approximate the bounded function values on its centers to a sufficiently fine finite set of levels. Thus there are finitely many functions \(f_1,\ldots,f_K\) such that for each \(f_\delta\), some \(f_k\) satisfies \(\sup_\theta|f_\delta(\theta)-f_k(\theta)|<\varepsilon\).

Each \(f_k\) is bounded and continuous, so weak convergence gives \(\int f_k\,d\pi_m\to\int f_k\,d\pi\). Since there are only finitely many \(k\), the maximum of these finitely many integral differences tends to zero. For any \(f_\delta\) and its chosen \(f_k\), both measures are probability measures, so

\left|\int f_\delta\,d\pi_m-\int f_\delta\,d\pi\right| \leq 2\varepsilon+ \left|\int f_k\,d\pi_m-\int f_k\,d\pi\right|.

Taking the supremum over \(\delta\), then the limit superior in \(m\), bounds it by \(2\varepsilon\). Since \(\varepsilon\) is arbitrary, the supremum tends to zero. \(\square\)

Convergence Theorem for Finite Bayes Problems

The theorem states conditions at the level of risks and rules. They apply, for example, when the rule space is compact under a suitable topology and the risk varies continuously with the parameter and the rule.

Theorem: Suppose \(\Theta\) is compact metric, \(\mathcal{D}\) is a compact metric space of rules, and \(R:\Theta\times\mathcal{D}\to\mathbb{R}\) is bounded and jointly continuous. Let \(\pi_m\) be finite-support priors converging weakly to \(\pi\). For each \(m\), let \(\mathcal{D}_m\subseteq\mathcal{D}\) be the finite-action rule class, and suppose there are maps \(Q_m:\mathcal{D}\to\mathcal{D}_m\) such that \(\sup_{\theta\in\Theta,\,\delta\in\mathcal{D}}|R(\theta,Q_m\delta)-R(\theta,\delta)|\leq\eta_m\), where \(\eta_m\to0\). If each finite problem attains its minimum, let \(\delta_m\) minimize \(r(\pi_m,\delta)\) over \(\mathcal{D}_m\). Then \(r(\pi,\delta_m)\) tends to the Bayes risk under \(\pi\). Every convergent subsequence of \(\delta_m\) has a limit that is Bayes for \(\pi\). If the Bayes rule is unique in \(\mathcal{D}\), the entire sequence converges to it.

Proof. Joint continuity on the compact product \(\Theta\times\mathcal{D}\) implies that the family \(R(\cdot,\delta)\) is bounded and equicontinuous. By the lemma,

\varepsilon_m=\sup_{\delta\in\mathcal{D}}|r(\pi_m,\delta)-r(\pi,\delta)|\longrightarrow 0.

Compactness of \(\mathcal{D}\) and continuity of \(r(\pi,\cdot)\) ensure that an original Bayes rule \(\delta^*\) exists. Since \(Q_m\delta^*\in\mathcal{D}_m\), optimality of \(\delta_m\) in the finite problem gives

r(\pi,\delta_m) \leq r(\pi_m,\delta_m)+\varepsilon_m \leq r(\pi_m,Q_m\delta^*)+\varepsilon_m \leq r(\pi,Q_m\delta^*)+2\varepsilon_m \leq r(\pi,\delta^*)+\eta_m+2\varepsilon_m.

On the other hand, \(\delta^*\) minimizes \(r(\pi,\cdot)\), so \(r(\pi,\delta_m)\geq r(\pi,\delta^*)\). The displayed inequalities prove convergence of the integrated risks to the original Bayes risk.

Now take any convergent subsequence \(\delta_{m_k}\to\bar{\delta}\). Continuity of \(r(\pi,\cdot)\) gives \(r(\pi,\delta_{m_k})\to r(\pi,\bar{\delta})\). The risk convergence just proved implies \(r(\pi,\bar{\delta})=r(\pi,\delta^*)\), so \(\bar{\delta}\) is Bayes. If the Bayes rule is unique, every convergent subsequence has the same limit \(\delta^*\). Compactness then implies the whole sequence converges to \(\delta^*\): otherwise a subsequence staying a fixed positive distance from \(\delta^*\) would itself have a convergent subsequence with a different limit. \(\square\)

Joint continuity also tracks risks at individual parameter values. Along any convergent subsequence \(\delta_{m_k}\to\bar{\delta}\), continuity gives \(R(\theta,\delta_{m_k})\to R(\theta,\bar{\delta})\) for each \(\theta\). In fact, the convergence is uniform in \(\theta\), because \(R\) is uniformly continuous on the compact product.

Worked Bayes Rules and Risk Comparisons

Worked Example: A Two-Point Normal Model with a Finite Action Grid

Let \(\theta\in\{-1,1\}\) have equal prior probabilities, observe \(X\mid\theta\sim N(\theta,1)\), and use squared-error loss \(L(\theta,a)=(a-\theta)^2\). First allow any action in \([-1,1]\). The likelihood ratio for \(\theta=1\) against \(\theta=-1\) at \(X=x\) is

\frac{\exp(-(x-1)^2/2)}{\exp(-(x+1)^2/2)}=\exp(2x).

The posterior probability of \(\theta=1\) is therefore \(p(x)=1/(1+\exp(-2x))\), and the posterior mean is \(2p(x)-1=\tanh x\). For squared-error loss, the posterior expected loss at action \(a\) is \(1+a^2-2a\tanh x\), minimized over unrestricted real actions at \(a=\tanh x\).

Now restrict actions to the finite grid \(\{-1,0,1\}\). The posterior expected losses at these actions are respectively \(2+2\tanh x\), \(1\), and \(2-2\tanh x\). Thus choose \(-1\) when \(\tanh x<-1/2\), choose \(0\) when \(|\tanh x|<1/2\), and choose \(1\) when \(\tanh x>1/2\), with either adjacent action permitted at a tie. Since \(\tanh x\) is increasing, the thresholds are \(x=\pm\operatorname{atanh}(1/2)=\pm\frac12\log 3\). At \(x=1\), \(\tanh(1)\approx0.7616\); action \(1\) has posterior loss \(2-2(0.7616)=0.4768\), less than the loss \(1\) at action \(0\). The finite-action rule is the grid action nearest the posterior mean, and refining the grid makes this action approximation increasingly accurate.

Worked Example: Checking the Action-Rounding Error

Consider any observation model with parameter \(\theta\in[0,1]\), action space \([0,1]\), and loss \(L(\theta,a)=(a-\theta)^2\). Use an action grid with maximum distance \(h\) from any point in \([0,1]\) to a grid point, and round each action to a nearest grid point. For actions \(a,b\in[0,1]\),

|L(\theta,a)-L(\theta,b)| =|a-b|\,|a+b-2\theta| \leq 2|a-b|.

The inequality holds because both \(a+b\) and \(2\theta\) lie in \([0,2]\). Consequently, if \(Q\delta\) rounds the action drawn under a rule \(\delta\), the loss changes by at most \(2h\) for every observation and parameter. Taking expectations preserves the bound: \(\sup_{\theta,\delta}|R(\theta,Q\delta)-R(\theta,\delta)|\leq2h\). Thus the action-grid error \(\eta\) in the theorem can be taken as \(2h\), and it vanishes as the grid spacing shrinks.

Why Continuity Conditions Matter

Worked Example: Weak Prior Convergence Can Give the Wrong Finite Bayes Rule

Let \(\Theta=[0,1]\), with no observation and actions \(\{0,1\}\). Define the bounded measurable risks by \(R(\theta,0)=1\) if \(\theta\) is irrational and \(0\) if \(\theta\) is rational; let \(R(\theta,1)=1-R(\theta,0)\). Under the uniform prior, the rational numbers have probability zero, so \(r(\pi,0)=1\) and \(r(\pi,1)=0\). The unique Bayes action is \(1\).

For each \(m\), take \(\pi_m\) to put equal mass on the \(m+1\) rational grid points \(j/m\), \(j=0,\ldots,m\). These priors converge weakly to the uniform distribution, as their integrals are Riemann sums for continuous functions. But \(r(\pi_m,0)=0\) and \(r(\pi_m,1)=1\) for every \(m\). Every finite-problem Bayes action is \(0\), which is not Bayes under the limiting prior. The risks are discontinuous in \(\theta\), so the equicontinuity condition fails. This example shows why weak convergence of priors alone does not justify convergence of Bayes rules.

Common Pitfall and Practical Use

A common mistake is to compute a Bayes rule for each finite approximation and infer that the rules must converge simply because the parameter and action grids become dense. Density of grids is not, by itself, a risk bound. The prior approximation must control integrated risks uniformly over candidate rules, and the action approximation must make the risk change uniformly small. Compactness then supplies convergent subsequences; continuity identifies their limits as Bayes rules.

When applying the theorem, check these points explicitly:

The finite priors converge weakly to the target prior.
Risks are bounded and vary continuously enough in the parameter and rule to give uniform integrated-risk control.
Every candidate rule can be approximated by a finite-action rule with a risk error tending uniformly to zero.
The finite optimization problems attain their minima, and the rule space has the compactness needed for subsequential limits.

If the limiting Bayes rule is not unique, the theorem guarantees that every cluster point is Bayes, not that the entire sequence converges. Different finite approximations can select different Bayes rules. If uniqueness holds, compactness upgrades the subsequential conclusion to convergence of the full sequence.

Check Your Understanding

Use the definitions and approximation result to answer the following.

What does a finite-support prior approximation replace, and what does a finite-action approximation restrict?
Why does weak convergence of priors alone not guarantee uniform convergence of integrated risks over rules?
In the squared-error rounding example, why is the risk error at most twice the action-grid distance?
What does the approximation theorem guarantee when the original Bayes rule is not unique?
In the rational-grid example, which regularity condition fails, and how does that failure affect the finite Bayes actions?