When Least-Favorable Priors Escape to Infinity

Two Different Questions About Attainment

On a noncompact parameter space, a sequence of priors can approach the largest possible Bayes risk while placing less and less probability in every bounded region. This is one meaning of a least-favorable prior sequence “escaping to infinity.” It does not, by itself, say whether a minimax decision rule exists. The prior asks whether a supremum over priors is attained; the rule asks whether an infimum over procedures is attained. These are separate questions.

To make that distinction precise, let $\Theta$ be a parameter space, let $\delta$ denote a decision rule, and let $R(\theta,\delta)$ be its risk at parameter value $\theta$. For a probability measure $\pi$ on $\Theta$, the Bayes risk is the smallest average risk under $\pi$. The minimax value is the smallest possible worst-case risk.

Definition: The Bayes risk of a prior $\pi$ is $$ r(\pi)=\inf_{\delta}\int_{\Theta}R(\theta,\delta)\,\pi(d\theta). $$ The minimax value is $$ V=\inf_{\delta}\sup_{\theta\in\Theta}R(\theta,\delta). $$ A rule $\delta^*$ is minimax if $\sup_{\theta}R(\theta,\delta^*)=V$. When $V=\sup_{\pi}r(\pi)$, a prior attaining this supremum is called a least-favorable prior. A sequence $(\pi_j)$ with $r(\pi_j)\to V$ is a least-favorable sequence, whether or not any prior attains $V$.

For every prior and rule, average risk cannot exceed worst-case risk. Taking the infimum over rules gives $r(\pi)\leq V$. Thus $\sup_\pi r(\pi)\leq V$, but equality requires additional conditions; it is not automatic in every decision problem. In the normal-mean model below, we verify equality directly. That lets us distinguish failure of prior attainment from failure of minimax-rule attainment without relying on a general minimax theorem.

A Normal-Mean Model with Escaping Priors

Suppose $X\mid\theta\sim N(\theta,\sigma^2)$, where $\theta\in\mathbb{R}$ and $\sigma^2>0$ is known. The action is an estimate of $\theta$, and the loss is squared error, so $R(\theta,\delta)=E_\theta[(\delta(X)-\theta)^2]$. The parameter space is noncompact. We first calculate the Bayes risks of a sequence of proper priors.

Worked Example: Diffuse Normal Priors Approach the Minimax Value

Take the proper prior $\pi_\tau=N(0,\tau^2)$, with $\tau^2>0$. Completing the square in the product of the prior density and the likelihood gives the posterior distribution $$ \theta\mid X=x\sim N\left(\frac{\tau^2}{\tau^2+\sigma^2}x,\frac{\tau^2\sigma^2}{\tau^2+\sigma^2}\right). $$ Under squared-error loss, the posterior mean minimizes conditional expected loss. Its posterior variance is constant in $x$, so the Bayes risk is $$ r(\pi_\tau)=\frac{\tau^2\sigma^2}{\tau^2+\sigma^2}. $$ As $\tau^2\to\infty$, this tends to $\sigma^2$. Each prior in the sequence is a proper probability distribution, but the sequence is not tight: for any fixed $M>0$, $$ \pi_\tau([-M,M])=2\Phi(M/\tau)-1\longrightarrow 0, $$ where $\Phi$ is the standard normal distribution function. Thus the priors approach the value while their mass leaves every fixed bounded interval.

The calculation gives a lower bound on the minimax value. For every rule $\delta$, its worst-case risk is at least its average risk under $\pi_\tau$, which in turn is at least the Bayes risk $r(\pi_\tau)$. Consequently $V\geq r(\pi_\tau)$ for every $\tau$, and letting $\tau^2\to\infty$ gives $V\geq\sigma^2$. On the other hand, the rule $\delta_0(X)=X$ has risk $$ R(\theta,\delta_0)=E_\theta[(X-\theta)^2]=\sigma^2 $$ for every $\theta$. Hence $V\leq\sigma^2$, so $V=\sigma^2$, and $\delta_0$ is minimax. In this model, the supremum of the Bayes risks equals the minimax value, but this sequence of priors has not shown whether a different proper prior might attain it. The next result settles that question.

Why No Proper Prior Attains the Value Here

The strict inequality for every proper prior is important: the Gaussian sequence is not merely one unsuccessful attempt to find a least-favorable prior. In fact, no proper prior on $\mathbb{R}$ has Bayes risk $\sigma^2$ in this model. The argument uses the conditional mean under an arbitrary proper prior and the marginal density of $X$.

Theorem: In the normal-mean model with squared-error loss, every proper prior $\pi$ on $\mathbb{R}$ has $r(\pi)<\sigma^2$. Therefore $\sup_\pi r(\pi)=\sigma^2=V$, but no proper least-favorable prior exists. At the same time, $\delta_0(X)=X$ is a minimax rule.

Proof. Let $m(x)=E[\theta\mid X=x]$ under $\pi$. This conditional mean exists for every $x$: the function $|\theta|\exp(-(x-\theta)^2/(2\sigma^2))$ is bounded as a function of $\theta$, so its integral against any probability measure is finite. The same reasoning applies to the conditional second moment. Conditional expectation minimizes squared-error risk, and the integrated risk of the rule $X$ is $\sigma^2$. The conditional squared-error decomposition therefore gives

\sigma^2 =E[(\theta-X)^2] =E[(\theta-m(X))^2]+E[(m(X)-X)^2].

The first term on the right is $r(\pi)$. In particular, $r(\pi)\leq\sigma^2$. If equality held, then $E[(m(X)-X)^2]=0$, so $m(X)=X$ with probability one.

Write $g(x)=\int \phi_\sigma(x-\theta)\,\pi(d\theta)$ for the marginal density of $X$, where $\phi_\sigma$ is the $N(0,\sigma^2)$ density. This density is positive everywhere. Differentiating under the integral is valid because the derivative of the normal density is bounded as a function of its argument, and gives

g'(x)=\int \frac{\theta-x}{\sigma^2}\phi_\sigma(x-\theta)\,\pi(d\theta) =\frac{g(x)}{\sigma^2}\bigl(m(x)-x\bigr).

The functions $g$ and $g'$ are continuous, and $g(x)>0$. Thus $m(x)-x$ is continuous. If $m(X)=X$ with probability one, positivity of the marginal density and continuity imply $m(x)=x$ for every $x$. The derivative identity then gives $g'(x)=0$ for every $x$, so $g$ is constant. A positive constant on $\mathbb{R}$ cannot integrate to one, contradicting that $g$ is a probability density. Equality is impossible, hence $r(\pi)<\sigma^2$ for every proper $\pi$. The diffuse priors already constructed have Bayes risks tending to $\sigma^2$, so their supremum is $\sigma^2$; and the constant-risk rule $X$ attains the minimax value. $\square$

Worked Example: A Two-Point Prior Has Strictly Smaller Bayes Risk

As a concrete proper prior, put probability $1/2$ at each of $\theta=-1$ and $\theta=1$, and take $\sigma^2=1$. Bayes’ rule gives $$ P(\theta=1\mid X=x)=\frac{1}{1+e^{-2x}}, \qquad m(x)=E[\theta\mid X=x]=\tanh(x). $$ The marginal density of $X$ is the equally weighted mixture of $N(-1,1)$ and $N(1,1)$, which is positive for every real $x$. The conditional squared-error decomposition yields $$ r(\pi)=1-E[(m(X)-X)^2] =1-E[(\tanh(X)-X)^2]. $$ At $x=1$, $\tanh(1)\neq 1$. By continuity, $(\tanh(x)-x)^2$ is positive on some interval around $1$; that interval has positive probability under the positive marginal density. Therefore $E[(\tanh(X)-X)^2]>0$, and $r(\pi)<1=V$. The prior’s Bayes risk is strictly below the minimax value even though the rule $X$ still has worst-case risk exactly $1$.

The Same Separation in a Discrete Game

The normal example involves observation and estimation. A small decision problem makes the role of escaping mass particularly transparent. It also shows that the phenomenon does not depend on the normal likelihood or squared-error loss.

Worked Example: A Matching Decision with No Least-Favorable Prior

Let $\Theta=\{1,2,\ldots\}$, let the action space also be the positive integers, and suppose the observation carries no information. Use loss $L(\theta,a)=1$ when $a\neq\theta$, and $L(\theta,a)=0$ when $a=\theta$. A randomized rule is a probability distribution $q=(q_1,q_2,\ldots)$ over actions. Its risk at $\theta$ is $R(\theta,q)=1-q_\theta$.

For every such $q$, $q_\theta\to0$ as $\theta\to\infty$, because the probabilities sum to one. Thus $\sup_\theta R(\theta,q)=1$, and every rule has worst-case risk $1$. The minimax value is $V=1$, attained by every rule.

For any proper prior $\pi=(\pi_1,\pi_2,\ldots)$, the best action chooses an index with largest prior mass. Such a largest mass exists: since the masses sum to one, their positive supremum cannot be approached only by infinitely many distinct indices, each carrying mass bounded away from zero. The Bayes risk is consequently $r(\pi)=1-\max_j\pi_j<1$. Now take $\pi_n$ uniform on $\{n,n+1,\ldots,2n-1\}$. Its largest atom is $1/n$, so $r(\pi_n)=1-1/n\to1$. For any fixed finite set of parameter values, $\pi_n$ assigns it probability zero once $n$ is sufficiently large. The Bayes risks approach $V$, but no proper prior attains it. Here, too, nonattainment on the prior side coexists with attainment on the rule side.

How to Read an Escaping Sequence

In the normal example, “escaping” means that the prior sequence places vanishing probability on every fixed compact interval; in the discrete example, it eventually leaves every fixed finite subset. Neither sequence converges to a proper prior that attains the limiting Bayes risk. The phrase does not mean that an ordinary prior assigns positive probability to a point called infinity. Rather, it describes how a sequence of probability distributions behaves relative to bounded regions of the original parameter space.

Keep three claims separate when analyzing a minimax problem:

$r(\pi_j)\to V$ says a sequence of priors approaches the value.
$\sup_\pi r(\pi)=V$ says the Bayes-risk supremum equals the minimax value.
Existence of a minimax rule says some rule actually attains the worst-case-risk infimum.

The normal model proves all three statements individually: the diffuse sequence gives the first, the lower and upper bounds give the second, and $\delta_0(X)=X$ gives the third. Yet no proper prior attains the Bayes-risk supremum. A sequence approaching a supremum cannot be treated as an attaining prior unless a suitable limit exists in the space of priors and the Bayes-risk functional behaves appropriately at that limit.

Common Mistake

Wrong reasoning: “The Bayes risks approach the minimax value, so the limiting prior is least favorable; if no least-favorable prior exists, a minimax rule cannot exist either.” This confuses convergence of values with existence of an admissible limiting prior, and it confuses prior attainment with rule attainment.

Correction: Check each attainment question directly. In the normal model, every proper prior has Bayes risk strictly less than $\sigma^2$, while the rule $X$ has risk $\sigma^2$ at every parameter value. The rule is minimax even though the prior supremum is not attained. Also verify the equality $\sup_\pi r(\pi)=V$ rather than assuming it merely because a sequence of Bayes risks has been constructed.

Key takeaway: On a noncompact parameter space, nearly least-favorable priors may lose their mass from every bounded region. This can prevent a proper prior from attaining the Bayes-risk supremum, but it does not imply that the minimax value is unattained by a decision rule.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

In the normal-mean model, why does the sequence $N(0,\tau^2)$ establish $V\geq\sigma^2$? State the inequality connecting worst-case risk and Bayes risk.
For a fixed $M>0$, what happens to the probability that a $N(0,\tau^2)$ prior lies in $[-M,M]$ as $\tau^2\to\infty$? What does this say about tightness?
Why does $r(\pi)<\sigma^2$ for every proper prior not contradict $\sup_\pi r(\pi)=\sigma^2$?
In the discrete matching game, calculate the Bayes risk of a prior uniform on $n$ parameter values.
Give one sentence distinguishing failure of least-favorable-prior attainment from failure of minimax-rule attainment.