Finite-Support Approximation of Priors

Why Approximate a Prior by One With Finite Support?

A prior can spread its probability over an entire parameter space, while a finite-support prior concentrates it on finitely many parameter values. This replacement is useful when studying Bayes risks on a compact parameter space: it turns an integral over many possible parameter values into a finite weighted sum. The key question is whether that replacement changes the Bayes risk by much.

The answer depends on how sensitively the risk of each decision rule varies with the parameter. If all rules share a common modulus of continuity, moving prior mass only a small distance changes every rule's integrated risk by a uniformly small amount. Taking the infimum over rules then preserves that error bound.

Setup and Definitions

Definition: Let $(\Theta,d)$ be a compact metric space, and let $\mathcal D$ be a nonempty class of decision rules. The risk of rule $\delta\in\mathcal D$ at parameter $\theta$ is $R(\theta,\delta)$. For a Borel probability measure $\pi$ on $\Theta$, its Bayes risk is $$ r(\pi)=\inf_{\delta\in\mathcal D}\int_\Theta R(\theta,\delta)\,\pi(d\theta). $$ A prior has finite support if it assigns probability one to a finite subset of $\Theta$.

We use the following assumptions. For every $\delta$, the function $\theta\mapsto R(\theta,\delta)$ is Borel measurable and nonnegative. In addition, there is a common modulus of continuity $\omega$: it is finite for sufficiently small $t>0$, satisfies $\omega(t)\to0$ as $t\downarrow0$, and

|R(\theta,\delta)-R(\theta',\delta)| \leq \omega\bigl(d(\theta,\theta')\bigr) \quad\text{for every }\delta\in\mathcal D,\ \theta,\theta'\in\Theta.

The same function $\omega$ must work for every rule. This uniformity is what will allow an error bound to remain valid after taking an infimum over rules. Under these assumptions, each risk is bounded on $\Theta$. Indeed, choose a small radius $t$ for which $\omega(t)$ is finite and cover compact $\Theta$ by finitely many balls of radius $t$. For fixed $\delta$, the risk at any point differs from its value at one of the ball centers by at most $\omega(t)$; the finitely many center values are finite. Thus each risk is bounded and Borel measurable, so its integral under every prior is finite. Pointwise finiteness alone would not ensure this integrability; boundedness here follows from compactness and the common modulus.

Definition: A finite $\eta$-net is a finite set $N_\eta=\{\theta_1,\ldots,\theta_k\}\subseteq\Theta$ such that every $\theta\in\Theta$ is within distance $\eta$ of at least one point in $N_\eta$. Given such a net, assign each parameter to a nearest net point, breaking ties by the smallest index. The resulting map $q_\eta:\Theta\to N_\eta$ satisfies $d(\theta,q_\eta(\theta))\leq\eta$.

Compactness guarantees a finite $\eta$-net for every $\eta>0$. The nearest-point assignment is Borel measurable: the set assigned to a center is described by finitely many comparisons of continuous distance functions, with strict or non-strict comparisons encoding the tie-breaking rule. Given a prior $\pi$, define its quantized prior $\pi_\eta$ by

\pi_\eta(A)=\pi\bigl(q_\eta^{-1}(A)\bigr),\qquad A\subseteq\Theta\text{ Borel}.

Equivalently, $\pi_\eta$ places at each net point the probability mass of the region assigned to it. It is a probability measure supported on the finite net.

The Bayes-Risk Approximation Bound

Theorem: Under the setup and assumptions above, for every prior $\pi$, every finite $\eta$-net, and its quantized prior $\pi_\eta$, $$ |r(\pi)-r(\pi_\eta)|\leq \omega(\eta). $$

Proof. Fix a rule $\delta$. Since $\pi_\eta$ is the distribution of $q_\eta(\theta)$ when $\theta$ has distribution $\pi$, the difference between the two integrated risks is bounded as follows:

\begin{aligned} \left|\int_\Theta R(\theta,\delta)\,\pi(d\theta) -\int_\Theta R(\theta,\delta)\,\pi_\eta(d\theta)\right| &=\left|\int_\Theta \{R(\theta,\delta)-R(q_\eta(\theta),\delta)\}\,\pi(d\theta)\right|\\ &\leq\int_\Theta |R(\theta,\delta)-R(q_\eta(\theta),\delta)|\,\pi(d\theta)\\ &\leq\int_\Theta\omega(\eta)\,\pi(d\theta) =\omega(\eta). \end{aligned}

The first equality uses the definition of the quantized prior; the final inequality uses $d(\theta,q_\eta(\theta))\leq\eta$ and the fact that a modulus is nondecreasing. The bound holds for every rule with the same error $\omega(\eta)$. If two families of real numbers differ termwise by at most $\varepsilon$, their infima differ by at most $\varepsilon$: each term in the first family is at least the corresponding term in the second minus $\varepsilon$, and at most that term plus $\varepsilon$. Applying this fact to the integrated risks, with $\varepsilon=\omega(\eta)$, proves the theorem. $\square$

Worked Example: Squared-Error Risk on an Interval

Let $\Theta=[0,1]$, with no informative observation, and let the action be $a\in[0,1]$. The loss and risk are $R(\theta,a)=(a-\theta)^2$. For any $a,\theta,\theta'\in[0,1]$,

|(a-\theta)^2-(a-\theta')^2| =|\theta-\theta'|\,|2a-\theta-\theta'| \leq 2|\theta-\theta'|.

Thus a common modulus is $\omega(t)=2t$. Take a uniform prior on $[0,1]$ and divide the interval into $m$ equal cells. Map each cell to its midpoint. The finite prior assigns mass $1/m$ to each midpoint $(j-\tfrac12)/m$, for $j=1,\ldots,m$. Each parameter moves by at most $1/(2m)$, so the theorem bounds the Bayes-risk error by $1/m$.

For squared-error loss, the Bayes action is the prior mean: expanding $E[(a-\theta)^2]$ around $E[\theta]$ shows that its unique minimum is at $a=E[\theta]$. Its minimum value is the prior variance. The uniform prior therefore has Bayes risk $1/12$. For $m=4$, the finite prior has support $1/8,3/8,5/8,7/8$, mean $1/2$, and variance

\frac{1}{4}\left[ \left(-\frac{3}{8}\right)^2+ \left(-\frac{1}{8}\right)^2+ \left(\frac{1}{8}\right)^2+ \left(\frac{3}{8}\right)^2 \right] =\frac{5}{64}.

The actual absolute error is $1/12-5/64=1/192$, which is below the general bound $1/4$. The bound guarantees control without requiring an exact Bayes-risk calculation.

Approximating the Supremum of Bayes Risks

The approximation theorem also has a consequence for optimizing over priors. Write $B=\sup_\pi r(\pi)$, where the supremum is over all Borel priors on $\Theta$, and $B_{\mathrm{fin}}=\sup_\pi r(\pi)$, where this second supremum is restricted to finite-support priors. Then

Theorem: Under the same assumptions, $B=B_{\mathrm{fin}}$. In particular, for every $\varepsilon>0$, there is a finite-support prior whose Bayes risk is at least $B-\varepsilon$.

Proof. Every finite-support prior is a prior, so $B_{\mathrm{fin}}\leq B$. Conversely, fix any prior $\pi$. For every $\eta>0$, its quantized prior has finite support and satisfies

r(\pi)\leq r(\pi_\eta)+\omega(\eta) \leq B_{\mathrm{fin}}+\omega(\eta).

Letting $\eta\downarrow0$ gives $r(\pi)\leq B_{\mathrm{fin}}$, and taking the supremum over $\pi$ yields $B\leq B_{\mathrm{fin}}$. Hence the two suprema are equal. To obtain the $\varepsilon$-claim, choose $\pi$ with $r(\pi)>B-\varepsilon/2$, then choose $\eta$ such that $\omega(\eta)<\varepsilon/2$. Its quantized prior has Bayes risk greater than $B-\varepsilon$. $\square$

The supremum is finite in this setup: choose any one rule $\delta_0$. Its risk is bounded on compact $\Theta$, so $r(\pi)\leq\int R(\theta,\delta_0)\,\pi(d\theta)\leq\sup_\theta R(\theta,\delta_0)<\infty$ for every prior. Nonnegativity also gives $r(\pi)\geq0$.

Worked Example: Two Actions and an Asymmetric Prior

Let $\Theta=[0,1]$, with two actions, 0 and 1, and no informative observation. Suppose their risks are $R(\theta,0)=\theta$ and $R(\theta,1)=1-\theta$. Both satisfy the common modulus $\omega(t)=t$. For any prior with mean $\mu$, the integrated risks are $\mu$ and $1-\mu$, so its Bayes risk is $\min(\mu,1-\mu)$.

Consider the prior with density $2\theta$ on $[0,1]$. Its mean is $\int_0^1 2\theta^2\,d\theta=2/3$, and its Bayes risk is $1/3$. Quantize using the two-cell midpoint net $\{1/4,3/4\}$, with cells $[0,1/2)$ and $[1/2,1]$. The cell masses are

\int_0^{1/2}2\theta\,d\theta=\frac14,\qquad \int_{1/2}^1 2\theta\,d\theta=\frac34.

The quantized mean is $(1/4)(1/4)+(3/4)(3/4)=5/8$. Its Bayes risk is $1-5/8=3/8$. The absolute error is $3/8-1/3=1/24$, below the theorem's bound $\omega(1/4)=1/4$. The example also shows that quantization need not preserve the Bayes risk exactly.

Worked Example: One Bernoulli Observation

Suppose $X\sim\operatorname{Bernoulli}(\theta)$, $\theta\in[0,1]$, and estimate $\theta$ with an estimator $\delta$ taking values $d_0=\delta(0)$ and $d_1=\delta(1)$ in $[0,1]$. Under squared-error loss its risk is

R(\theta,\delta)=\theta(d_1-\theta)^2+(1-\theta)(d_0-\theta)^2.

This class of risks has common modulus $\omega(t)=3t$. To see this, write $g_i(\theta)=(d_i-\theta)^2$, so $0\leq g_i(\theta)\leq1$. For $\theta,\theta'\in[0,1]$, the change in the Bernoulli weights contributes at most $|\theta-\theta'|$, since $|g_1(\theta)-g_0(\theta)|\leq1$. Also,

|g_i(\theta)-g_i(\theta')| =|\theta-\theta'|\,|2d_i-\theta-\theta'| \leq2|\theta-\theta'|.

The change in the loss terms, averaged with the Bernoulli weights, is therefore at most $2|\theta-\theta'|$. Adding these bounds gives the claimed modulus.

Start with the uniform prior and quantize it to the two midpoints $1/4$ and $3/4$, each with mass $1/2$. The net radius is $1/4$, so the Bayes-risk error is at most $3/4$. Under the original prior, $P(X=1)=P(X=0)=1/2$. Given $X=1$, the posterior density is $2\theta$, with mean $2/3$ and variance $1/18$; given $X=0$, it is $2(1-\theta)$, with mean $1/3$ and variance $1/18$. Squared-error posterior risk is minimized by the posterior mean, so the Bayes risk is $1/18$.

Under the finite prior, the marginal probability of $X=1$ is also $1/2$. Given $X=1$, the posterior probabilities at $1/4$ and $3/4$ are $1/4$ and $3/4$, respectively, because the likelihoods are $1/4$ and $3/4$. Its posterior mean is $5/8$, and its variance is

\frac14\left(\frac14-\frac58\right)^2+ \frac34\left(\frac34-\frac58\right)^2 =\frac{3}{64}.

For $X=0$, the posterior weights reverse and the variance is again $3/64$. Thus the finite prior's Bayes risk is $3/64$, and the actual error is $1/18-3/64=5/576$, within the general bound.

Why the Common Modulus Matters

A common mistake is to argue that if every individual rule has a continuous risk function, then a sufficiently fine net must approximate the Bayes risk uniformly well. That reasoning overlooks the infimum over rules. Continuity for each rule separately does not supply one error bound that works across the entire class; rules can vary more and more sharply as the rule changes. The proof needs a common modulus, or another condition that directly controls the integrated-risk error uniformly over the rules. Without it, the termwise comparison cannot be passed through the infimum with a fixed error.

A second pitfall is to infer integrability under every prior merely from finite risk at each parameter value. A finite-valued measurable function on a compact space can still have an infinite integral under some prior. Here, the common modulus and compactness ensure that each rule's risk is bounded; alternatively, boundedness or integrability under the priors being considered must be assumed explicitly. These checks matter whenever prior approximation is used to make claims about Bayes risks or approximate least-favorable priors.

Key takeaway: A finite net moves every parameter by at most its radius. If every decision rule's risk changes by at most the same modulus at that distance, then quantizing any prior changes its Bayes risk by at most that modulus—and finite-support priors attain the same supremum of Bayes risks as unrestricted priors.

Check Your Understanding

Use the assumptions and proof above to answer these questions.

Why must the modulus of continuity be common to all decision rules to obtain the stated bound after taking an infimum?
If a finite net has radius $\eta$ and $\omega(t)=Ct$, what is the resulting bound on the absolute Bayes-risk error?
In the squared-error example, what is the Bayes action under a prior, and why does the prior variance give its Bayes risk?
For the two-action example, what is the Bayes risk when the prior mean is $0.4$?
Why does a finite-valued Borel risk function not, by itself, guarantee a finite integral under every prior?