Finite-Support Priors in Complete-Class Arguments

Why Finite-Support Priors Matter

A prior turns a risk function into one number: its average risk. That number defines a linear functional on the set of possible risk functions, so a prior can identify a supporting boundary of that set. When the parameter space is finite, this connection is exact: a suitable supporting normal is just a vector of prior probabilities. With a continuous parameter space, finite-support priors can often approximate the same boundary information, but the approximation requires conditions on the risk functions.

There is an important qualification. Not every supporting hyperplane corresponds to a prior. Prior weights must be nonnegative and have sum one. A supporting normal with a negative coordinate cannot be interpreted as prior probabilities merely because it is a linear functional on risk vectors.

Definition: For a decision rule $\delta$, its risk at parameter value $\theta$ is $R(\theta,\delta)=E_\theta[L(\theta,\delta(X))]$, when this expectation exists. For a probability distribution $\pi$ on the parameter space, the integrated risk is $$ r_\pi(\delta)=\int R(\theta,\delta)\,\pi(d\theta). $$ A finite-support prior is a probability distribution of the form $\pi=\sum_{j=1}^m w_j\delta_{\theta_j}$, where $w_j\geq 0$, $\sum_{j=1}^m w_j=1$, and $\delta_{\theta_j}$ denotes unit mass at $\theta_j$. Its integrated risk is $\sum_{j=1}^m w_jR(\theta_j,\delta)$.

Finite Risk Vectors: When a Supporting Normal Is a Prior

Suppose the parameter space consists of $m$ points, $\theta_1,\ldots,\theta_m$. A rule then has a risk vector $x=(R(\theta_1,\delta),\ldots,R(\theta_m,\delta))$ in $\mathbb{R}^m$. A prior with weights $w_1,\ldots,w_m$ evaluates this vector by the dot product $w\cdot x$. Thus, if a nonzero supporting normal $w$ to a lower risk boundary has nonnegative coordinates, it can be normalized into prior weights.

Theorem: Let $C\subseteq\mathbb{R}^m$ be a set of risk vectors, and let $x^*\in C$. Suppose a nonzero vector $a\in\mathbb{R}^m$ satisfies $a\cdot x\geq a\cdot x^*$ for every $x\in C$, and suppose $a_j\geq 0$ for every $j$. Then $w_j=a_j/\sum_{k=1}^m a_k$ are prior weights on $\{\theta_1,\ldots,\theta_m\}$, and $x^*$ minimizes integrated risk under that prior. The nonnegativity condition is essential.

To verify the claim, set $s=\sum_{k=1}^m a_k$. Since $a$ is nonzero and coordinatewise nonnegative, $s>0$. Dividing the supporting inequality by $s$ gives $$ \sum_{j=1}^m w_jx_j\geq \sum_{j=1}^m w_jx_j^*, \qquad w_j\geq 0,\qquad \sum_{j=1}^m w_j=1. $$ The left side is the integrated risk of the rule with risk vector $x$, so $x^*$ minimizes that risk. This identifies precisely which supporting normals yield priors; a general supporting normal need not do so.

Worked Example: Normalizing a Valid Supporting Normal

Suppose the available risk vectors are $x^*=(1,3)$ and $y=(3,1)$. Consider the normal $a=(2,1)$. At $x^*$, the linear functional equals $2(1)+1(3)=5$; at $y$, it equals $2(3)+1(1)=7$. Thus $a\cdot x\geq a\cdot x^*$ for both available vectors, and $a$ supports the lower boundary at $x^*$.

Both coordinates are nonnegative, and their sum is $3$. The associated prior weights are $w=(2/3,1/3)$. The integrated risk at $x^*$ is $2/3+1=5/3$, whereas at $y$ it is $2+1/3=7/3$. Hence $x^*$ minimizes integrated risk under this prior, as the supporting inequality predicts.

Now consider a different supporting normal for the same two-point set: $a=(-1,0)$. Its values are $-1$ at $x^*$ and $-3$ at $y$, so it supports the set at $y$ when the inequality is oriented to minimize this functional. But it cannot be normalized into prior weights: its first coordinate is negative. This functional is not an integrated risk under any prior on the two parameter points.

Approximating a Prior on a Continuous Parameter Space

On a continuous parameter space, a finite grid can approximate integrated risks if risk functions vary in a controlled way with $\theta$. The key condition is uniform continuity over the decision rules being considered. Without it, a grid that approximates the prior itself need not approximate the integrated risks uniformly across rules.

Theorem: Let $\Theta$ be a compact metric space with metric $d$, let $\pi$ be a probability distribution on $\Theta$, and suppose every risk function $R(\cdot,\delta)$ is bounded and measurable. Assume there is a common modulus of continuity $\omega$, with $\omega(h)\to 0$ as $h\downarrow 0$, such that $$ |R(\theta,\delta)-R(\theta',\delta)|\leq \omega(d(\theta,\theta')) $$ for every rule $\delta$ and all $\theta,\theta'\in\Theta$. For every $h>0$, there is a finite-support prior $\pi_h$ such that $$ \sup_\delta\left|\int R(\theta,\delta)\,\pi_h(d\theta)-\int R(\theta,\delta)\,\pi(d\theta)\right|\leq\omega(h). $$

Proof. Cover $\Theta$ by finitely many measurable sets $A_1,\ldots,A_m$, each of diameter at most $h$, disjointly assigning any overlapping parts. For each set with positive $\pi$-probability choose $\theta_j\in A_j$, and assign it weight $w_j=\pi(A_j)$. Sets of zero probability can be omitted. The weights are nonnegative and sum to one, so $\pi_h=\sum_j w_j\delta_{\theta_j}$ is a finite-support prior.

For any rule $\delta$, the difference between its integrated risks is $$ \left|\sum_{j=1}^m\int_{A_j}\left\{R(\theta_j,\delta)-R(\theta,\delta)\right\}\pi(d\theta)\right| \leq\sum_{j=1}^m\int_{A_j}\omega(h)\,\pi(d\theta) =\omega(h). $$ The bound does not depend on $\delta$, so taking the supremum over rules proves the stated result. $\square$

The uniform bound also controls the best integrated risk. Write $b(\pi)=\inf_\delta r_\pi(\delta)$. If $|r_{\pi_h}(\delta)-r_\pi(\delta)|\leq\varepsilon$ for every rule, then $b(\pi)-\varepsilon\leq b(\pi_h)\leq b(\pi)+\varepsilon$. Indeed, each integrated risk under either prior is within $\varepsilon$ of the other, so taking infima preserves those inequalities. Therefore a continuous-prior supporting functional can be approximated by finite-support prior functionals, uniformly over the rule class, when the stated conditions hold.

This is an approximate conclusion, not an assertion that the finite prior supports the exact same boundary point. For example, suppose $r^*$ minimizes integrated risk under $\pi$. If all rules’ integrated risks change by at most $\varepsilon$, then the excess risk of $r^*$ under $\pi_h$, relative to the minimum under $\pi_h$, is at most $2\varepsilon$: its risk can increase by at most $\varepsilon$, while the minimum can decrease by at most $\varepsilon$. If instead $r^*$ has a strict integrated-risk advantage of at least $\gamma>0$ over every competitor under $\pi$, that advantage remains strict under $\pi_h$ whenever $2\varepsilon<\gamma$.

Worked Example: Bounding Grid Error for a Family of Risks

Let $\Theta=[0,1]$, let $\pi$ be uniform on this interval, and consider the risk functions $R_a(\theta)=a\theta$ for $a\in[0,2]$. Partition $[0,1]$ into $m$ intervals of width $1/m$, and put the mass of each interval at its midpoint. The resulting prior $\pi_m$ has $m$ support points and weights $1/m$.

For each interval, the distance from its midpoint to any point in that interval is at most $1/(2m)$. Also, $|R_a(\theta)-R_a(\theta')|=a|\theta-\theta'|\leq 2|\theta-\theta'|$, so a common modulus is $\omega(h)=2h$. The theorem therefore gives $$ \sup_{0\leq a\leq 2}|r_{\pi_m}(a)-r_\pi(a)|\leq \frac{1}{m}. $$

The exact calculation is even sharper. Under the uniform prior, $r_\pi(a)=a\int_0^1\theta\,d\theta=a/2$. The midpoint average of the $m$ grid points is $1/2$, so $r_{\pi_m}(a)=a/2$ as well. Thus the actual error is zero for this family, consistent with the general bound. The bound applies even when a particular risk family does not have this exact cancellation.

A Continuous-Parameter Decision Example

Consider a problem with parameter space $\Theta=[0,1]$, no informative observation, action set $\{0,1\}$, and squared-error loss $L(\theta,a)=(a-\theta)^2$. A randomized rule chooses action $1$ with probability $q\in[0,1]$ and action $0$ with probability $1-q$. Randomization makes its risk the corresponding mixture of the two action risks.

Worked Example: A Symmetric Finite Prior Exposes a Risk Boundary

The risk of action $0$ is $\theta^2$, and the risk of action $1$ is $(1-\theta)^2$. Hence the risk of the randomized rule is $$ R(\theta,q)=(1-q)\theta^2+q(1-\theta)^2. $$ Under the uniform prior, $\int_0^1\theta^2\,d\theta=1/3$ and $\int_0^1(1-\theta)^2\,d\theta=1/3$. Therefore $r_\pi(q)=1/3$ for every $q\in[0,1]$. Each available rule minimizes integrated risk, so this functional supports the risk set at each of its risk functions.

For a finite approximation, take the $m$ midpoint values $\theta_j=(j-\tfrac12)/m$, each with prior weight $1/m$. The grid is symmetric: $1-\theta_j=\theta_{m+1-j}$. Directly, $$ \frac{1}{m}\sum_{j=1}^m\theta_j^2 =\frac{1}{m^3}\sum_{j=1}^m(j-\tfrac12)^2 =\frac{4m^2-1}{12m^2}. $$ By symmetry, the average of $(1-\theta_j)^2$ has the same value. Thus every randomized rule has finite-prior integrated risk $(4m^2-1)/(12m^2)$, independent of $q$. The finite prior also supports the risk set at every rule.

The finite-prior value differs from the uniform-prior value by $$ \frac{1}{3}-\frac{4m^2-1}{12m^2}=\frac{1}{12m^2}, $$ which tends to zero. In this example symmetry gives exact support for every finite grid, while the displayed calculation also verifies convergence to the continuous-prior value. In less symmetric problems, finite priors generally approximate the supporting functional rather than reproducing it exactly.

What the Approximation Does—and Does Not—Establish

Finite-support priors are useful because they reduce integrated-risk comparisons to finitely many parameter values. In finite risk-vector geometry, a nonnegative supporting normal yields an exact prior after normalization. For continuous parameter spaces, compactness and a common continuity bound allow a prior to be replaced by finite atomic measures with uniformly small error in integrated risk. That uniformity is what lets the approximation apply across a whole class of decision rules rather than just one selected rule.

Exact finite-dimensional representation: a supporting normal represents a prior only when its coordinates are nonnegative and it is nonzero.
Approximation on a continuous space: a common modulus of continuity gives a uniform error bound for finite-support priors.
Strict comparisons: a strict gap survives if the approximation error is smaller than half the gap.

A common mistake is to claim that every supporting hyperplane is a Bayes-risk hyperplane. For example, the functional $x\mapsto -x_1$ on the risk vectors $(0,1)$ and $(1,0)$ is minimized at $(1,0)$, but its coefficient vector $(-1,0)$ cannot be a prior. The correction is to check the orientation and signs of the normal: for minimization of integrated risk, the normal must be coordinatewise nonnegative and nonzero before normalization.

A second pitfall is to approximate a prior by a grid and assume that all integrated risks are therefore close. Approximation of the prior alone is not enough for uniform control over arbitrary risk functions. The common continuity condition in the theorem supplies the missing bound; without appropriate regularity, a risk function may vary sharply between grid points.

Check Your Understanding

Use the definitions and conditions above to answer the following questions.

Why must a nonzero supporting normal have nonnegative coordinates before it can be normalized into prior weights?
For a finite-support prior with weights $w_1,\ldots,w_m$, write the integrated risk of a rule in terms of its risk vector.
Which regularity condition lets a finite grid approximate integrated risks uniformly over a class of rules?
If all integrated risks change by at most $\varepsilon$, why can the best integrated risk change by at most $\varepsilon$, rather than $2\varepsilon$?
In the squared-error example, why does the symmetric midpoint prior give the same integrated risk for actions $0$ and $1$?