A Finite Least-Favorable-Prior Calculation Workshop

From a Finite Model to a Least-Favorable Prior

A finite decision problem is small enough that its risks can often be calculated exactly, yet rich enough to show the essential logic of least-favorable-prior analysis. The calculation has several connected parts: list the rules and their risks, vary the prior weights, find a Bayes rule at a maximizing prior, and check that the resulting rule controls risk at every parameter value. The last check matters: a prior can make a rule perform well on average while leaving its risk large at a parameter value that the prior barely weights or omits.

We will work with finite parameter and observation spaces, a finite action set, and a specified loss. A rule may be randomized: after observing the data, it may select an action according to a probability distribution. The risk is the expected loss under the sampling distribution at a fixed parameter value.

Definition: For parameter values $\theta_1,\ldots,\theta_k$, a rule $\delta$ has risk $R(\theta_i,\delta)=E_{\theta_i}[L(\theta_i,\delta(X))]$, with the expectation also including any randomization in the rule. A prior $\pi=(\pi_1,\ldots,\pi_k)$ has $\pi_i\geq 0$ and $\sum_i\pi_i=1$. The prior risk of $\delta$ is $\sum_i\pi_iR(\theta_i,\delta)$, and the Bayes risk is $$ r(\pi)=\inf_{\delta}\sum_{i=1}^k\pi_iR(\theta_i,\delta). $$ A least-favorable prior maximizes $r(\pi)$ over priors on the parameter space.

For a finite problem, a risk matrix is a convenient record of risks. Its rows index parameter values and its columns index rules. Randomized rules may add risks not present as columns for deterministic rules; those risks must still be evaluated when checking whether a candidate is minimax.

Worked Example: Optimizing the Prior in a Binary Testing Problem

Worked Example: A Binary Observation and a Least-Favorable Prior

Let $\theta\in\{0,1\}$, and observe $X\in\{0,1\}$, with $P_0(X=0)=0.8$, $P_0(X=1)=0.2$, $P_1(X=0)=0.4$, and $P_1(X=1)=0.6$. The action $a\in\{0,1\}$ is a decision about the parameter, and the loss is $L(\theta,a)=1$ if $a\ne\theta$ and $0$ otherwise.

First list all four deterministic rules. Write a rule by the actions it takes at $X=0$ and $X=1$. The resulting risk matrix is

\begin{array}{c|cccc} & (0,0)&(1,1)&(1,0)&(0,1)\\ \hline \theta=0&0&1&0.8&0.2\\ \theta=1&1&0&0.6&0.4 \end{array}

For example, rule $(0,1)$ makes an error under $\theta=0$ only when $X=1$, giving risk $0.2$; under $\theta=1$ it errs only when $X=0$, giving risk $0.4$.

Let $q=P(\theta=1)$, so $P(\theta=0)=1-q$. The prior risks of the four deterministic rules, in the same column order, are $q$, $1-q$, $0.8(1-q)+0.6q=0.8-0.2q$, and $0.2(1-q)+0.4q=0.2+0.2q$. Thus the Bayes risk is the smallest of these four quantities. For $q\leq 2/3$, the risk of rule $(0,1)$ is at most $0.2+0.2(2/3)=1/3$, so $r(q)\leq 1/3$. For $q\geq 2/3$, rule $(1,1)$ has prior risk $1-q\leq 1/3$, again giving $r(q)\leq 1/3$.

At $q=2/3$, rule $(1,1)$ has prior risk $1/3$, and rule $(0,1)$ also has prior risk $(1/3)(0.2)+(2/3)(0.4)=1/3$. No rule can have prior risk below $1/3$: at $X=0$, the prior-weighted error contributions for actions 0 and 1 are respectively $(2/3)(0.4)=4/15$ and $(1/3)(0.8)=4/15$; at $X=1$, choosing action 1 has contribution $(1/3)(0.2)=1/15$, smaller than action 0's contribution $(2/3)(0.6)=2/5$. The minimum total contribution is therefore $4/15+1/15=1/3$. This proves $r(2/3)=1/3$, so the bounds above establish that $q=2/3$ is least favorable.

At $X=0$ the two actions tie under this prior. Choose action 1 there with probability $s$, and choose action 1 with certainty at $X=1$. Under $\theta=0$, the risk is $0.8s+0.2$; under $\theta=1$, it is $0.4(1-s)$. Equating these risks gives $0.8s+0.2=0.4-0.4s$, hence $s=1/6$. Substitution gives risk $1/3$ under each parameter value. This randomized rule is Bayes for the least-favorable prior, since it randomizes only between tied actions at $X=0$ and chooses the lower-contribution action at $X=1$.

Why the Risk Check Completes the Argument

The prior calculation alone establishes an average-risk lower bound, not the performance of a proposed rule at every parameter value. A short criterion makes the needed two-sided check precise. It applies to finite parameter spaces and does not require a general minimax theorem.

Theorem: Suppose $\pi$ is a prior and $\delta^*$ is Bayes for $\pi$, with Bayes risk $B=r(\pi)$. If $R(\theta,\delta^*)\leq B$ for every parameter value and $R(\theta,\delta^*)=B$ whenever $\pi(\theta)>0$, then $\pi$ is least favorable and $\delta^*$ is minimax, both with value $B$.

Proof. For any rule $\delta$, its maximum risk is at least its prior-average risk, and its prior-average risk is at least the Bayes risk:

\sup_{\theta}R(\theta,\delta) \geq \sum_{\theta}\pi(\theta)R(\theta,\delta) \geq r(\pi)=B.

Thus no rule has maximum risk below $B$. The assumed upper bound gives $\sup_\theta R(\theta,\delta^*)\leq B$, so $\delta^*$ attains the minimum possible maximum risk. Moreover, for any other prior $\nu$, using $\delta^*$ as a candidate rule in the infimum defining $r(\nu)$ gives

r(\nu)\leq\sum_{\theta}\nu(\theta)R(\theta,\delta^*)\leq B.

The prior $\pi$ attains $B$, so it maximizes Bayes risk and is least favorable. The equality condition on the support ensures that the prior average of $\delta^*$'s risk is $B$; it also makes explicit how the prior and the rule balance one another. $\square$

In the binary example, $B=1/3$, both parameter values have positive prior probability, and the randomized rule has risk $1/3$ at both. The theorem therefore confirms both least favorability and minimaxity. In this finite calculation, the saddle-point conditions have a direct interpretation: the rule minimizes average risk for the chosen prior, while that prior averages the rule's risk at its maximum value.

Worked Example: An Uninformative Experiment

Worked Example: Prior Optimization When the Data Carry No Information

Suppose $\theta\in\{0,1\}$, the observation has the same distribution under both parameter values, and the action is an estimate in $\{0,1\}$, with zero-one loss. Since the data distribution does not depend on $\theta$, observing $X$ cannot change which action minimizes prior risk. Put $q=P(\theta=1)$. Choosing action 0 has prior risk $q$; choosing action 1 has prior risk $1-q$. Consequently,

r(q)=\min\{q,1-q\}.

For $q\leq 1/2$, this equals $q$, which increases up to $1/2$. For $q\geq1/2$, it equals $1-q$, which decreases from $1/2$. Hence the maximum Bayes risk is $1/2$, attained at the equal prior. At that prior both actions are Bayes. A rule choosing each action with probability $1/2$, independently of the observation, has risk $1/2$ at $\theta=0$ and at $\theta=1$. Its maximum risk equals the Bayes-risk lower bound, verifying the saddle-point conditions and minimaxity.

Worked Example: A Least-Favorable Prior on Two Endpoints

Worked Example: Three Parameter Values and Absolute-Error Loss

There is no informative observation. The parameter and action spaces are both $\{0,1,2\}$, and the loss is $L(\theta,a)=|\theta-a|$. Write the prior probabilities as $p_0,p_1,p_2$. The expected losses of actions 0, 1, and 2 are respectively $p_1+2p_2$, $p_0+p_2$, and $2p_0+p_1$. Therefore the Bayes risk is the minimum of these three values.

For every prior, action 1 gives prior risk $p_0+p_2\leq1$, so $r(\pi)\leq1$. Now take $p_0=p_2=1/2$, $p_1=0$. The three action risks are all 1: $p_1+2p_2=1$, $p_0+p_2=1$, and $2p_0+p_1=1$. Thus the Bayes risk is 1, proving that this endpoint prior is least favorable.

To obtain a rule whose risk is 1 at every parameter value, choose action 0 or action 2 with probability $1/2$ each. At $\theta=0$, its risk is $(1/2)0+(1/2)2=1$; at $\theta=1$, it is $(1/2)1+(1/2)1=1$; and at $\theta=2$, it is $(1/2)2+(1/2)0=1$. Both actions used by the rule are Bayes under the endpoint prior, and its risk equals the Bayes risk everywhere. The theorem therefore establishes minimaxity with value 1, including at the parameter value $\theta=1$, which has zero prior mass.

Common Pitfall: Treating the Prior Average as a Maximum-Risk Result

A common mistake is to find a prior with a large Bayes risk and conclude that a particular Bayes rule is minimax without checking its risks outside the prior average. The Bayes risk gives a lower bound on the smallest achievable maximum risk, but it does not by itself show that a candidate rule reaches that bound. The correction is to calculate the candidate's risk at every parameter value and verify that none exceeds the bound. If the prior omits a parameter value, that check is especially important: its risk is not constrained by equality on the prior's support.

In a finite calculation, a reliable sequence is to enumerate deterministic rules and form their risk matrix; express prior risks as functions of the prior weights; maximize the minimum prior risk; identify the Bayes actions at the maximizing prior; and then check the risks of the chosen, possibly randomized, rule over the entire parameter space. When actions tie, randomization can be useful because it may balance the risks without changing Bayes optimality.

Check Your Understanding

Use the definitions and calculations in this workshop to answer the following.

In the binary testing example, why does the prior-risk bound for $q\leq2/3$ use rule $(0,1)$, while the bound for $q\geq2/3$ uses rule $(1,1)$?
At $q=2/3$ and $X=0$, calculate the prior-weighted error contribution for each action. Why may a Bayes rule randomize there?
State the two checks in the theorem that turn a Bayes rule into a minimax rule and its prior into a least-favorable prior.
In the absolute-error example, why is the risk at $\theta=1$ essential even though the least-favorable prior assigns it probability zero?
For an uninformative binary experiment under zero-one loss, what is the Bayes risk as a function of $q$, and where is it maximized?