Verifying a Candidate Prior by Risk Bounds

From a Candidate Prior to a Minimax Rule

A prior can suggest a useful decision rule, but its Bayes risk alone does not show that the rule performs well for every parameter value. The decisive check is whether the rule’s risk is bounded above by that same Bayes risk throughout the parameter space. When it is, a lower bound from the prior and an upper bound from the rule meet, identifying the minimax risk.

This approach is especially useful in finite problems: calculate the Bayes rule and its average risk, then inspect its risk at every state. The check is global, not restricted to parameter values receiving positive prior probability.

Definition: A statistical decision problem specifies a parameter space \(\Theta\), an observation \(X\) with distribution depending on \(\theta\in\Theta\), an action space \(\mathcal A\), and a loss \(L(\theta,a)\). A decision rule \(\delta\) maps observations to actions; randomized rules are allowed when stated. Its risk is \(R(\theta,\delta)=E_\theta[L(\theta,\delta(X))]\). For a prior \(\pi\), the Bayes risk is \(r(\pi)=\inf_\delta \int_\Theta R(\theta,\delta)\,\pi(d\theta)\). A rule attaining this infimum is a Bayes rule for \(\pi\). The minimax risk is \(V=\inf_\delta\sup_{\theta\in\Theta}R(\theta,\delta)\).

Assume the risks and prior averages used below are well-defined, and that the candidate Bayes rule attains its Bayes risk. For example, these conditions hold in a finite decision problem with finite losses. A prior’s Bayes risk is an average risk; it is not, by itself, a bound on the risk at each parameter value.

The Risk-Bound Criterion

For any rule, its maximum risk must be at least its average risk under any prior. Since the Bayes risk is the smallest such average risk, every rule faces a lower bound determined by the candidate prior. If that lower bound matches a uniform upper bound for the candidate Bayes rule, the rule is minimax.

Theorem: Let \(\pi\) be a prior with finite Bayes risk \(r(\pi)\), and let \(\delta_\pi\) be a Bayes rule for \(\pi\). Suppose \(R(\theta,\delta_\pi)\le r(\pi)\) for every \(\theta\in\Theta\). Then \(\sup_{\theta\in\Theta}R(\theta,\delta_\pi)=V=r(\pi)\), so \(\delta_\pi\) is minimax and \(\pi\) is least favorable.

Here, “least favorable” means that the prior attains the largest Bayes risk among priors, where that largest value equals the minimax risk. The conclusion follows directly from the lower and upper bounds; no minimax interchange is needed.

Calculate the Bayes risk.
Find the prior-averaged risk of a Bayes rule and verify that it is \(r(\pi)\).

Bound the risk globally.
Show that \(R(\theta,\delta_\pi)\le r(\pi)\) for every \(\theta\), including values outside the prior’s support.

Match the bounds.
The prior gives a lower bound on minimax risk, and the rule gives an upper bound. Both equal \(r(\pi)\).

Proof. For any rule \(\delta\), its maximum risk is at least its prior-averaged risk. That average is at least the Bayes risk:

\sup_{\theta\in\Theta}R(\theta,\delta) \ge \int_\Theta R(\theta,\delta)\,\pi(d\theta) \ge r(\pi).

Taking the infimum over \(\delta\) gives \(V\ge r(\pi)\). The assumed bound on \(\delta_\pi\) gives \(V\le\sup_\theta R(\theta,\delta_\pi)\le r(\pi)\). Hence \(V=r(\pi)\). Since \(\delta_\pi\) is a Bayes rule, its prior-averaged risk is \(r(\pi)\); therefore its maximum risk is at least \(r(\pi)\), and the assumed upper bound makes it exactly \(r(\pi)\). Finally, every prior’s Bayes risk is at most \(V\), by the same lower-bound argument. Since \(\pi\)’s Bayes risk equals \(V\), it is least favorable. \(\square\)

Useful consequence: Under the theorem’s risk bound, \(R(\theta,\delta_\pi)=r(\pi)\) for \(\pi\)-almost every \(\theta\). Indeed, the risk is at most \(r(\pi)\) everywhere, while its average under \(\pi\) equals \(r(\pi)\). A strict inequality on a set of positive prior probability would make that average smaller.

For a finite parameter space, this consequence says that every state with positive prior mass must have risk exactly \(r(\pi)\). States assigned zero prior mass need not have equal risk, but they must still meet the upper bound.

Worked Verifications

Worked Example: A Two-State Classification Problem

Let \(\Theta=\{0,1\}\), and observe \(X\in\{0,1\}\). Suppose \(P_\theta(X=\theta)=3/4\) and \(P_\theta(X\ne\theta)=1/4\). The action is a classification \(\hat\theta\in\{0,1\}\), with loss \(L(\theta,\hat\theta)=\mathbf{1}\{\theta\ne\hat\theta\}\). Take the prior with mass \(1/2\) on each state.

For either observation \(x\), Bayes’ rule gives posterior probability \(3/4\) to \(\theta=x\) and \(1/4\) to the other state. Under zero-one loss, choosing the more probable state minimizes posterior expected loss, so \(\delta_\pi(x)=x\).

The rule’s risk at \(\theta=0\) is the probability of observing \(1\), namely \(1/4\). At \(\theta=1\), its risk is the probability of observing \(0\), also \(1/4\). Thus the prior average of the risks is \((1/2)(1/4)+(1/2)(1/4)=1/4\). This is the Bayes risk because the rule is Bayes. Its risk is everywhere \(1/4\), so the theorem shows that the rule is minimax and the equal-weight prior is least favorable, with minimax risk \(1/4\).

Worked Example: Three-State Classification

Let \(\Theta=\{1,2,3\}\), with observations \(X\in\{1,2,3\}\) satisfying \(P_\theta(X=\theta)=0.6\) and \(P_\theta(X=x)=0.2\) for each \(x\ne\theta\). Use zero-one classification loss and the uniform prior.

For any observed \(x\), each likelihood is specified by the channel: the likelihood under \(\theta=x\) is \(0.6\), and under either other state it is \(0.2\). Since the prior weights are equal, the posterior probability of \(\theta=x\) is \(0.6/(0.6+0.2+0.2)=0.6\); each other state has posterior probability \(0.2\). Therefore a Bayes rule is \(\delta_\pi(x)=x\).

For each true state \(\theta\), the rule is wrong exactly when \(X\ne\theta\), which has probability \(0.2+0.2=0.4\). Its risk is thus \(0.4\) at all three states. The Bayes risk is its uniform average: \((0.4+0.4+0.4)/3=0.4\). Since the rule’s risk never exceeds \(0.4\), the theorem establishes minimaxity and least favorability of the uniform prior.

Worked Example: A Prior That Fails, and One That Works

Suppose there is no informative observation, \(\Theta=\{-1,0,1\}\), actions are real numbers \(a\), and loss is squared error \(L(\theta,a)=(a-\theta)^2\). A rule is therefore a fixed action. First try the uniform prior.

Under the uniform prior, the Bayes action minimizes \(\{(a+1)^2+a^2+(a-1)^2\}/3\). Expanding gives \((3a^2+2)/3=a^2+2/3\), which is minimized at \(a=0\). Its Bayes risk is \(2/3\). But the risk of action \(0\) is \(1\) at \(\theta=-1\), \(0\) at \(\theta=0\), and \(1\) at \(\theta=1\). Since \(1>2/3\), the required risk bound fails, so this calculation does not verify the uniform prior as least favorable.

Now put probability \(1/2\) on each endpoint, \(-1\) and \(1\). The Bayes action minimizes \(\{(a+1)^2+(a-1)^2\}/2=a^2+1\), so it is again \(a=0\), with Bayes risk \(1\). Across the full parameter space \(\{-1,0,1\}\), its risks are \(1,0,1\), all at most \(1\). The criterion now proves minimaxity and least favorability.

For a direct check of the minimax value, any action \(a\) has average endpoint risk \(\{(a+1)^2+(a-1)^2\}/2=a^2+1\ge1\). Its maximum risk over the parameter space is at least this average, hence at least \(1\); action \(0\) attains maximum risk \(1\).

What the Check Establishes—and What It Does Not

The criterion is a certificate: once the Bayes risk and the full risk bound are verified, the minimax conclusion follows. It does not say that every prior’s Bayes rule will satisfy the bound. A failed bound means this particular certificate has not succeeded; it does not, by itself, prove that the prior is not least favorable or that its Bayes rule is not minimax.

Two checks deserve special attention. First, calculate the Bayes risk using the prior average of the rule’s risks, or by a valid posterior calculation; do not substitute the largest risk for the average. Second, inspect every parameter value. A prior concentrated on a subset can still produce a candidate rule whose risk is too large at a state outside that subset, as the uniform-prior attempt in the squared-error example illustrates.

When a finite problem is organized as a risk table, the candidate’s prior weights average the risks down the rows or across the states, depending on the table’s convention. The theorem requires a separate comparison of each state’s risk with the resulting Bayes risk. The average identifies the lower bound; the largest statewise risk determines whether the proposed rule supplies the matching upper bound.

Key takeaway: A candidate prior is verified by two linked calculations: its Bayes rule has Bayes risk \(r(\pi)\), and that rule’s risk is at most \(r(\pi)\) at every parameter value. Together they force the minimax risk, the rule’s maximum risk, and the candidate’s Bayes risk to be equal.

Check Your Understanding

Use the risk-bound criterion to organize your reasoning.

Why is the maximum risk of any rule at least its average risk under a fixed prior?
In the three-state classification example, what is the posterior probability of the observed state, and why does that determine the Bayes action?
Why must the risk bound be checked at parameter values outside the candidate prior’s support?
For the uniform prior in the squared-error example, identify the Bayes risk and the state where the proposed bound fails.
If a candidate Bayes rule has risk at most its Bayes risk everywhere, why must its maximum risk equal—not merely be less than—its Bayes risk?

Verifying a Candidate Prior by Risk Bounds

What You'll Learn