Tightness Criteria for Approximate Least-Favorable Priors

Why Tightness Matters

A sequence of priors can have Bayes risks approaching the largest possible value while its probability mass moves farther and farther out in the parameter space. Without a convergent subsequence, there may be no limiting prior at which that value is achieved. Tightness is the condition that prevents this escape and makes a compactness argument possible.

The key conclusion needs two ingredients. Tightness gives a weakly convergent subsequence; continuity of the risk makes the Bayes-risk functional behave well along that subsequence. With an appropriate minimax equality, these facts show that the limit is least favorable. Neither ingredient can simply be omitted.

Definitions and the Tightness Criterion

Definition: Let $\Theta$ be a Polish space, and let $\pi_n$ be Borel probability measures on $\Theta$. The sequence $(\pi_n)$ is tight if, for every $\varepsilon>0$, there is a compact set $K\subseteq\Theta$ such that $$ \sup_n \pi_n(K^c)<\varepsilon. $$ A prior $\pi$ is least favorable if its Bayes risk equals the supremum of Bayes risks over all priors.

The uniformity over $n$ matters: it requires one compact set to capture almost all the mass of every prior in the sequence. For $\Theta=\mathbb{R}$, tightness is equivalent to requiring that, for every $\varepsilon>0$, some $M<\infty$ satisfies $\sup_n\pi_n(\{|\theta|>M\})<\varepsilon$. On a Polish space, Prokhorov’s theorem says that a family of probability measures is relatively compact for weak convergence if and only if it is tight. In particular, a tight sequence has a weakly convergent subsequence.

Worked Example: Checking Tightness on the Real Line

Consider the priors $\pi_n=N(0,1+1/n)$, for $n\geq 1$. To check tightness, take any $\varepsilon>0$. Their variances are at most $2$, so Chebyshev’s inequality gives, for each $M>0$, $$ \sup_{n\geq 1}\pi_n(\{|\theta|>M\}) \leq \sup_{n\geq 1}\frac{1+1/n}{M^2} \leq \frac{2}{M^2}. $$ Choose $M>\sqrt{2/\varepsilon}$. Then the interval $[-M,M]$ is compact and the displayed tail bound is less than $\varepsilon$. Thus the sequence is tight. In contrast, for $\pi_n=\delta_n$, every compact subset of $\mathbb{R}$ omits all sufficiently large $n$, so its complement has $\pi_n$-probability $1$ for such $n$. That sequence is not tight.

Now specify the decision-theoretic quantities. Let $\delta$ denote a decision rule and let $R(\theta,\delta)$ be its risk at parameter value $\theta$. For a prior $\pi$, define its Bayes risk by $$ r(\pi)=\inf_\delta \int_\Theta R(\theta,\delta)\,\pi(d\theta), $$ and define the minimax value by $$ V=\inf_\delta\sup_{\theta\in\Theta}R(\theta,\delta). $$ The central result below uses the minimax equality $V=\sup_\pi r(\pi)$. It is stated as an assumption: it does not follow from tightness alone.

Near-least-favorable sequence: A sequence $(\pi_n)$ is near-least-favorable if $r(\pi_n)\to V$. Equivalently, when $r(\pi_n)\leq V$, it is enough that $r(\pi_n)\geq V-\varepsilon_n$ for some $\varepsilon_n\to 0$.

The Tightness Theorem

Theorem: Suppose $\Theta$ is Polish, $R(\theta,\delta)$ is bounded and continuous in $\theta$ for every decision rule $\delta$, and $V=\sup_\pi r(\pi)$. If a near-least-favorable sequence of priors is tight, then some subsequence converges weakly to a prior $\pi^*$, and $\pi^*$ is least favorable: $r(\pi^*)=V$.

The boundedness hypothesis is understood uniformly in the parameter for each rule: for each $\delta$, there is a finite constant $C_\delta$ with $|R(\theta,\delta)|\leq C_\delta$ for all $\theta$. It ensures that weak convergence applies to each risk function as a bounded continuous test function. The proof does not require the infimum over decision rules to be attained.

Extract a weak limit.
By tightness and Prokhorov’s theorem, choose a subsequence $\pi_{n_k}$ and a prior $\pi^*$ such that $\pi_{n_k}\xrightarrow{w}\pi^*$.

Pass each fixed rule’s risk to the limit.
For any fixed $\delta$, bounded continuity gives $\int R(\theta,\delta)\,\pi_{n_k}(d\theta)\to\int R(\theta,\delta)\,\pi^*(d\theta)$.

Use the infimum carefully.
The infimum of continuous functions of $\pi$ is upper semicontinuous. Hence the limiting Bayes risk is at least the limiting near-optimal values.

Here is the complete justification of the last step. For each fixed $\delta$, the map $\pi\mapsto\int R(\theta,\delta)\,\pi(d\theta)$ is continuous under weak convergence. The Bayes risk is the infimum of these maps. For any real $a$, the set $\{\pi:r(\pi)<a\}$ is the union over $\delta$ of the open sets $\{\pi:\int R(\theta,\delta)\,\pi(d\theta)<a\}$. It is therefore open, which is precisely upper semicontinuity of $r$. Thus $$ r(\pi^*)\geq\limsup_{k\to\infty}r(\pi_{n_k})=V. $$ Since $V=\sup_\pi r(\pi)$, we also have $r(\pi^*)\leq V$. Therefore $r(\pi^*)=V$, as claimed. $\square$

Worked Example: A Tight Sequence Approaching a Least-Favorable Prior

Consider a decision problem with no data, parameter space $\Theta=\mathbb{R}$, action $a\in[-1,1]$, and loss $L(\theta,a)=(a-\cos\theta)^2$. The risk is bounded and continuous in $\theta$. Let $\pi^*=\tfrac12\delta_0+\tfrac12\delta_\pi$. Under this prior, the target value $\cos\theta$ is $1$ or $-1$, each with probability $1/2$. For any action $a$, the prior expected loss is $$ \frac12(a-1)^2+\frac12(a+1)^2=a^2+1. $$ Its minimum over $a\in[-1,1]$ is $1$, at $a=0$. More generally, for any prior, the best constant action is the prior mean of $\cos\theta$, and its Bayes risk is $\operatorname{Var}_\pi(\cos\theta)\leq 1$. Thus $\pi^*$ attains the largest Bayes risk.

For $n\geq 1$, set $\pi_n=(1-1/n)\pi^*+(1/n)\delta_n$. For every bounded continuous function $h$, $$ \int h\,d\pi_n=(1-1/n)\int h\,d\pi^*+\frac{h(n)}{n} \longrightarrow \int h\,d\pi^*, $$ so $\pi_n\xrightarrow{w}\pi^*$. The sequence is also tight: for any $\varepsilon>0$, choose $N$ with $1/N<\varepsilon$, and take a compact interval containing $0,\pi,1,\ldots,N$. For $n>N$, at most mass $1/n<\varepsilon$ lies outside it; for $n\leq N$, all the mass lies inside it. By the theorem, the limiting prior is least favorable. This example also shows that a small amount of mass can drift outward without destroying tightness when the amount is uniformly controlled.

When Mass Escapes to Infinity

Failure of tightness removes the subsequence guarantee, but it does not by itself prove that no least-favorable prior exists. It does mean that the tightness theorem cannot supply one. The following decision problem shows that escape can coincide with genuine nonattainment.

Worked Example: Near-Least-Favorable Priors with No Attaining Limit

Again take no data, actions $a\in[-1,1]$, and parameter space $\mathbb{R}$, but now use the loss $L(\theta,a)=(a-\tanh\theta)^2$. The range of $\tanh\theta$ is $(-1,1)$. For any fixed action $a$, the supremum risk is $$ \sup_{\theta\in\mathbb{R}}(a-\tanh\theta)^2=(1+|a|)^2. $$ Its infimum over $a\in[-1,1]$ is $1$, attained at $a=0$, so $V=1$.

For a prior $\pi$, the best constant action is $E_\pi[\tanh\Theta]$, which belongs to $[-1,1]$, and its Bayes risk is $\operatorname{Var}_\pi(\tanh\Theta)$. Since $|\tanh\Theta|<1$ at every finite $\Theta$, $$ \operatorname{Var}_\pi(\tanh\Theta) =E_\pi[(\tanh\Theta)^2]-\big(E_\pi[\tanh\Theta]\big)^2<1. $$ The strict inequality follows because $1-(\tanh\Theta)^2>0$ everywhere, so its expectation is positive. Thus no prior has Bayes risk $1$.

Nevertheless, the priors $\pi_m=\tfrac12\delta_{-m}+\tfrac12\delta_m$ have mean target zero and Bayes risk $\tanh^2(m)\to 1$. They are not tight: any compact set is contained in some bounded interval, which omits both support points for all sufficiently large $m$. Their Bayes risks approach $V$, but no least-favorable prior exists. Here mass escapes toward parameter values whose target values approach, but never equal, the two endpoints.

Worked Example: Normal Priors Drifting Away

Let $\pi_n=N(n,1)$ on $\mathbb{R}$. To see that no compact set captures most of the mass uniformly, fix $M>0$. For $n>M$, write $\Theta=n+Z$, where $Z\sim N(0,1)$. Then $$ \pi_n([-M,M])=P(-M-n\leq Z\leq M-n) \leq P(|Z|\geq n-M) \leq \frac{1}{(n-M)^2}, $$ where the final inequality is Chebyshev’s inequality. This probability tends to zero as $n\to\infty$, so the family is not tight.

The same argument applies to every subsequence. In particular, no subsequence can converge weakly to a probability measure: if one did, choose a bounded open interval with limit probability greater than $1/2$; the Portmanteau theorem would force its probabilities along the subsequence to have limit inferior at least that large, contradicting the bound just obtained. This calculation is a compactness diagnostic; by itself, it does not assert that these priors are near-least-favorable for any particular decision problem.

Common Pitfall and Practical Use

Incorrect reasoning: “The Bayes risks converge to $V$, so the priors must converge to a least-favorable prior.” Bayes-risk convergence alone gives no convergence of the priors. The $\tfrac12\delta_{-m}+\tfrac12\delta_m$ sequence above has near-maximal Bayes risk and escapes every compact set.
Correction: Establish tightness first. Then extract a weakly convergent subsequence and use bounded continuity of the risks to obtain upper semicontinuity of Bayes risk. Finally, use $V=\sup_\pi r(\pi)$ to identify the limit as least favorable.
Another limitation: A non-tight sequence does not prove nonattainment. It only blocks this compactness argument. A least-favorable prior might exist for other reasons.

In applications, tail bounds for the priors are often the decisive step. On $\mathbb{R}$, a uniform bound making $\sup_n\pi_n(\{|\theta|>M\})$ small for large $M$ establishes tightness. If the tails do not shrink uniformly, inspect whether the escaping mass is essential to the limiting Bayes risk, as it is in the $\tanh$ example. This separates a technical gap in a proof from a real failure of attainment.

Key takeaway: Tightness of near-least-favorable priors, together with bounded continuous risks and the minimax equality, yields a weakly convergent subsequence whose limit is least favorable. Without tightness, near-optimal Bayes risks may be approached only by priors that escape to infinity.

Check Your Understanding

Use the definitions and results above to answer these questions.

State the uniform compact-set condition that defines tightness for a sequence of priors.
Why does bounded continuity of each fixed rule’s risk imply continuity of its prior-integrated risk under weak convergence?
In the tightness theorem, where is the minimax equality $V=\sup_\pi r(\pi)$ used?
For the $\tanh$ loss example, why does every individual prior have Bayes risk strictly less than $1$, even though a sequence of Bayes risks tends to $1$?
Does failure of tightness alone establish that a least-favorable prior does not exist? Explain.