Bayes Risk Sequences Approaching Minimax Values

Why a Sequence of Priors Can Matter

A least-favorable prior is useful because it turns a worst-case problem into a Bayes problem: its Bayes risk reaches the minimax value. But the supremum of Bayes risks need not be attained by any single prior. In that case, there is no least-favorable prior, even though priors can still come arbitrarily close to the value that such a prior would have achieved.

The key distinction is between attaining a supremum and approaching a supremum. When the minimax value equals the supremum Bayes risk, the definition of supremum supplies a sequence of priors with Bayes risks converging to that value. This conclusion does not require the sequence of priors to converge, and it does not by itself make the corresponding Bayes rules minimax.

Bayes Risk and the Minimax Value

Let \(\Theta\) be a parameter space, let \(X\) have distribution \(P_\theta\) under parameter \(\theta\), and let \(\delta\) be a possibly randomized decision rule. Suppose the loss \(L(\theta,a)\) is measurable and nonnegative. Assume the risk \(R(\theta,\delta)=E_\theta[L(\theta,\delta(X))]\) is measurable in \(\theta\). For a prior probability measure \(\pi\) on \(\Theta\), define its Bayes risk as the smallest prior-averaged risk among the allowed rules. The infimum need not be achieved.

r(\pi)=\inf_{\delta}\int_{\Theta}R(\theta,\delta)\,\pi(d\theta), \qquad V=\inf_{\delta}\sup_{\theta\in\Theta}R(\theta,\delta).

Here \(V\) is the minimax value. We consider the finite-value case \(V<\infty\), and take the supremum of \(r(\pi)\) over all prior probability measures under consideration. Nonnegative loss ensures that the prior risk integrals are well-defined, including when they are infinite.

Definition: A prior \(\pi^\star\) is least favorable (for this minimax problem) if it attains the supremum of Bayes risks and that supremum equals the minimax value: \(r(\pi^\star)=\sup_\pi r(\pi)=V\). If the supremum equals \(V\) but no prior attains it, the minimax value has no least-favorable prior.

The first fact is a weak-duality bound: a prior average cannot exceed the worst risk of the same rule. Taking infima preserves the corresponding comparison between Bayes and minimax values.

Theorem (weak duality): For every prior \(\pi\), \(r(\pi)\leq V\). Consequently, \(\sup_\pi r(\pi)\leq V\).

Proof. For every rule \(\delta\), nonnegativity and measurability make its prior-averaged risk well-defined, and an average is no greater than the supremum of the values being averaged:

\int_{\Theta}R(\theta,\delta)\,\pi(d\theta) \leq \sup_{\theta\in\Theta}R(\theta,\delta).

Taking the infimum over \(\delta\) on both sides gives

r(\pi) =\inf_\delta\int_{\Theta}R(\theta,\delta)\,\pi(d\theta) \leq \inf_\delta\sup_{\theta\in\Theta}R(\theta,\delta) =V.

This holds for each \(\pi\), so taking the supremum over priors proves the second assertion. \(\square\)

The Approximating-Prior Result

Theorem: Suppose \(V<\infty\) and \(\sup_\pi r(\pi)=V\). For every \(\varepsilon>0\), there is a prior \(\pi_\varepsilon\) such that \(V-\varepsilon<r(\pi_\varepsilon)\leq V\). In particular, there is a sequence of priors \((\pi_n)\) for which \(r(\pi_n)\to V\). The sequence can be taken constant if a least-favorable prior exists.

Proof. Since \(V\) is the supremum of the set of Bayes risks, it is an upper bound for that set. If no prior had Bayes risk greater than \(V-\varepsilon\), then \(V-\varepsilon\) would also be an upper bound, contradicting that the supremum is \(V\). Thus some prior \(\pi_\varepsilon\) satisfies \(r(\pi_\varepsilon)>V-\varepsilon\). Weak duality gives \(r(\pi_\varepsilon)\leq V\). Taking \(\varepsilon=1/n\) gives

V-\frac{1}{n}<r(\pi_n)\leq V,

which implies \(r(\pi_n)\to V\). If a least-favorable prior exists, use it for every \(n\). \(\square\)

The theorem is an existence result, not a recipe for locating a useful prior in a particular model. It also needs the equality \(\sup_\pi r(\pi)=V\); weak duality alone gives only the upper bound. The sequence may have no limit as a sequence of probability measures. The conclusion concerns its Bayes risks.

For each \(\pi_n\), a Bayes rule need not exist. Nevertheless, because \(r(\pi_n)\) is a finite infimum, for any \(\eta_n>0\) there is a rule \(\delta_n\) whose prior-averaged risk is less than \(r(\pi_n)+\eta_n\). For instance, choose \(\eta_n=1/n\). Then

V-\frac{1}{n} < \int_{\Theta}R(\theta,\delta_n)\,\pi_n(d\theta) < r(\pi_n)+\frac{1}{n} \leq V+\frac{1}{n}.

This gives rules with prior-averaged risks approaching \(V\). It is a statement about averages under the selected priors, not a bound on \(\sup_\theta R(\theta,\delta_n)\).

Worked Examples

Worked Example: Approaching the Value Without a Least-Favorable Prior

There is no informative observation, the parameter space is \(\Theta=(0,1)\), the action is any real number \(a\), and the loss is \(L(\theta,a)=(a-\theta)^2\). Find the minimax value and a sequence of priors whose Bayes risks approach it.

For a fixed action \(a\), the supremum risk is \(\sup_{0<\theta<1}(a-\theta)^2\). Values of \(\theta\) can approach either endpoint, so this supremum is \(\max\{a^2,(1-a)^2\}\). Its minimum over \(a\in\mathbb R\) is \(1/4\), attained at \(a=1/2\). Thus \(V=1/4\).

For any prior \(\pi\), regard \(\theta\) as a random variable with distribution \(\pi\). Expanding around its mean gives

E_\pi[(a-\theta)^2] = (a-E_\pi[\theta])^2+\operatorname{Var}_\pi(\theta).

The minimizing action is \(a=E_\pi[\theta]\), and the Bayes risk is \(\operatorname{Var}_\pi(\theta)\). Since \(0<\theta<1\),

\operatorname{Var}_\pi(\theta) \leq E_\pi[(\theta-\tfrac12)^2] <\frac14.

The strict inequality holds because \((\theta-\tfrac12)^2<1/4\) at every point of \((0,1)\). Therefore no prior has Bayes risk \(1/4\).

For \(0<\varepsilon<1/2\), let \(\pi_\varepsilon\) put probability \(1/2\) at each of \(\varepsilon\) and \(1-\varepsilon\). Its mean is \(1/2\), and its variance, hence its Bayes risk, is

r(\pi_\varepsilon) = \frac12(\varepsilon-\tfrac12)^2+ \frac12(1-\varepsilon-\tfrac12)^2 = (\tfrac12-\varepsilon)^2.

As \(\varepsilon\) decreases to zero, this converges to \(1/4=V\), although every prior in the sequence has Bayes risk strictly below \(V\). This is a concrete case where an approximating sequence exists but a least-favorable prior does not.

Worked Example: A Least-Favorable Prior and a Constant Sequence

Again there is no informative observation. Now \(\Theta=\{1,2\}\), the action set is \(\{1,2\}\), and the loss is \(1\) when the action differs from the parameter and \(0\) otherwise. Let a randomized rule choose action \(1\) with probability \(q\). Its risks are \(R(1,\delta)=1-q\) and \(R(2,\delta)=q\).

The largest risk is \(\max(1-q,q)\). It is minimized at \(q=1/2\), giving \(V=1/2\). Let a prior assign probability \(p\) to parameter \(1\) and \(1-p\) to parameter \(2\). The prior-averaged risk of the rule is

$$ p(1-q)+(1-p)q=p+q(1-2p). $$

Minimizing over \(0\leq q\leq1\) yields \(r(\pi)=\min(p,1-p)\): choose \(q=0\) if \(p<1/2\), \(q=1\) if \(p>1/2\), and any \(q\) if \(p=1/2\). The maximum Bayes risk is \(1/2\), attained at \(p=1/2\). Thus this prior is least favorable, and the constant sequence \(\pi_n=\pi\) has \(r(\pi_n)=V\) for every \(n\).

Worked Example: Near-Maximal Bayes Risk Does Not Ensure Small Worst-Case Risk

Use the same two-parameter decision problem. For each integer \(n\geq1\), assign prior probability \(p_n=1/2+1/(4n)\) to parameter \(1\). Since \(p_n>1/2\), the Bayes rule chooses action \(1\) with certainty. Its risk is \(0\) at parameter \(1\) and \(1\) at parameter \(2\). Therefore its worst-case risk is \(1\), not \(V=1/2\).

Its prior-averaged risk and the Bayes risk under this prior are both

r(\pi_n)=1-p_n =\frac12-\frac{1}{4n} \longrightarrow \frac12=V.

So the priors have Bayes risks approaching the minimax value, while their Bayes rules have worst-case risk \(1\) for every \(n\). A near-maximal Bayes risk is not, on its own, a worst-case performance guarantee for a Bayes rule.

What the Sequence Does—and Does Not—Guarantee

Approximating priors are useful when the relevant value is known to be \(V\), but no prior attains it. They provide Bayes problems whose optimal average risks are as close to \(V\) as desired. If exact Bayes rules do not exist, approximate Bayes rules still give prior-averaged risks close to \(V\), by choosing their approximation error to tend to zero.

A common mistake is to reason that because \(r(\pi_n)\to V\), the rules that are Bayes under \(\pi_n\) must be approximately minimax. The third example disproves this: the average is close to \(V\), but the maximum risk stays at \(1\). A prior places weights on parameter values; its average can be close to \(V\) without controlling risk at every parameter.

The correction is to keep the two claims separate. The theorem guarantees a sequence of priors with nearly maximal Bayes risks. Any claim about the worst-case risk of associated rules requires additional argument or assumptions. Likewise, the existence of a sequence does not imply that its priors converge or that any limiting prior attains the supremum.

Key takeaway: If \(V=\sup_\pi r(\pi)<\infty\), then for every positive tolerance there is a prior whose Bayes risk lies within that tolerance of \(V\). Attainment is a separate question, and closeness of prior-averaged risk is not by itself closeness of worst-case risk.

Check Your Understanding

Use the definitions and arguments above to answer the following.

Why does weak duality imply \(r(\pi)\leq V\) for each prior, rather than merely \(\sup_\pi r(\pi)\leq V\)?
If \(\sup_\pi r(\pi)=V\) but no prior attains the supremum, what property of the supremum guarantees a prior within \(\varepsilon\) of \(V\)?
In the open-interval squared-error example, why is the Bayes risk under a prior equal to the prior variance?
In the two-parameter example, calculate the Bayes risk when the prior probability of parameter \(1\) is \(p=0.3\). Which action is Bayes?
Why does convergence of Bayes risks to \(V\) not, by itself, establish that the corresponding rules have worst-case risks converging to \(V\)?