Computing Bayes Risk by Posterior-Minimum Decomposition

Why Decompose Bayes Risk by Observation?

A prior assigns probabilities to parameter values before the data are observed. Once an observation arrives, the decision can respond to that observation; the same action need not be best for every possible data value. Bayes risk accounts for this flexibility by choosing, at each observation, an action that minimizes posterior expected loss. The resulting finite-sample calculation is a sum over possible observations, not a single minimization that ignores the data.

This decomposition is useful both for evaluating a specified prior and for comparing candidate priors. Its key computational advantage is that the decision rule can be optimized one observation at a time. In finite models, this often turns a search over all decision rules into a short list of posterior minimizations.

Setup and Posterior-Minimum Formula

Let the parameter set \(\Theta=\{\theta_1,\ldots,\theta_k\}\) and observation set \(\mathcal X\) be finite. Under parameter \(\theta\), write \(p_\theta(x)=P_\theta(X=x)\). A prior is a probability vector \(\pi=(\pi_1,\ldots,\pi_k)\), where \(\pi_i=P(\theta=\theta_i)\). Let \(\mathcal A\) be an action set and let \(L(\theta_i,a)\) be a finite loss for choosing action \(a\) when the parameter is \(\theta_i\). A nonrandomized decision rule \(\delta\) assigns an action \(\delta(x)\) to each \(x\in\mathcal X\).

The prior-predictive probability of observing \(x\) is \(m_\pi(x)=\sum_i\pi_i p_{\theta_i}(x)\). If \(m_\pi(x)>0\), Bayes' rule gives posterior probabilities \(P(\theta_i\mid x)=\pi_i p_{\theta_i}(x)/m_\pi(x)\). When \(m_\pi(x)=0\), the observation has probability zero under the prior, so its posterior need not be defined.

Definition: The risk of a rule \(\delta\) at \(\theta_i\) is \(R(\theta_i,\delta)=\sum_{x\in\mathcal X}p_{\theta_i}(x)L(\theta_i,\delta(x))\). Its Bayes risk under \(\pi\) is \(r(\pi)=\inf_\delta\sum_i\pi_iR(\theta_i,\delta)\). At an observation \(x\) with \(m_\pi(x)>0\), the posterior risk of action \(a\) is \(\sum_iP(\theta_i\mid x)L(\theta_i,a)\).

The formula below assumes that, at every \(x\) with \(m_\pi(x)>0\), the posterior expected loss attains its minimum over \(\mathcal A\). This holds automatically if \(\mathcal A\) is finite; it also holds in many familiar problems with a continuous action set, such as squared-error estimation.

Theorem: Under the finite-model setup and minimum-attainment condition,

r(\pi)=\sum_{x\in\mathcal X}\min_{a\in\mathcal A}\sum_{i=1}^k\pi_i p_{\theta_i}(x)L(\theta_i,a).

Equivalently, each term with \(m_\pi(x)>0\) is \(m_\pi(x)\) times the minimum posterior expected loss at \(x\). If \(m_\pi(x)=0\), every \(\pi_i p_{\theta_i}(x)\) is zero, so the displayed minimum weighted loss is zero and contributes nothing.

Why the Decomposition Holds

For any rule \(\delta\), rearranging the finite sums in its prior-averaged risk gives

\sum_i\pi_iR(\theta_i,\delta) =\sum_{x\in\mathcal X}\sum_i\pi_i p_{\theta_i}(x)L(\theta_i,\delta(x)).

For each fixed \(x\), the action \(\delta(x)\) cannot have smaller weighted loss than the minimum over all actions. Summing these pointwise lower bounds shows that every rule has Bayes risk at least as large as the right side of the theorem. To attain that bound, choose at each positive-probability observation an action \(a_x\) minimizing the weighted loss, and set \(\delta(x)=a_x\). There are finitely many observations, so these choices define a rule. At zero-probability observations choose any action; their contribution is zero. This rule attains the sum of the pointwise minima, proving the equality. \(\square\)

Randomization does not lower the minimum in this calculation. A randomized action at a fixed \(x\) has expected weighted loss equal to an average of the losses of its possible actions, and an average cannot be below the smallest of those losses.

Worked Example: Binary Classification With Unequal Prior Weight

Let \(\Theta=\{0,1\}\), let the observation be \(X\in\{0,1\}\), and suppose \(P(X=1\mid\theta=1)=0.8\) while \(P(X=1\mid\theta=0)=0.2\). Use the zero-one loss \(L(\theta,a)=1\) if \(a\ne\theta\), and \(0\) otherwise. Write \(q=P(\theta=1)\), so \(P(\theta=0)=1-q\). At each observation, the minimum weighted loss is the smaller of the two weighted probabilities of making an error.

At \(x=1\), choosing action 0 incurs weighted loss \(0.2(1-q)\); choosing action 1 incurs \(0.8q\). The contribution is therefore \(\min\{0.8q,0.2(1-q)\}\). At \(x=0\), choosing action 0 incurs \(0.2q\), while choosing action 1 incurs \(0.8(1-q)\). The contribution is \(\min\{0.2q,0.8(1-q)\}\). Thus

r(q)=\min\{0.8q,0.2(1-q)\}+\min\{0.2q,0.8(1-q)\}.

The first minimum switches at \(0.8q=0.2(1-q)\), which gives \(q=0.2\). The second switches at \(0.2q=0.8(1-q)\), which gives \(q=0.8\). Consequently,

r(q)= \begin{cases} q, & 0\le q\le 0.2,\\ 0.2, & 0.2\le q\le 0.8,\\ 1-q, & 0.8\le q\le 1. \end{cases}

For example, at \(q=0.5\), each observation contributes \(0.1\), so the Bayes risk is \(0.2\). The formula also shows that every prior weight between \(0.2\) and \(0.8\) has that same risk; the maximum over this family of priors is \(0.2\), but the maximizing prior is not unique.

Posterior Losses Can Simplify the Calculation

When \(m_\pi(x)>0\), the weighted minimum for \(x\) can be factored as

\min_a\sum_i\pi_i p_{\theta_i}(x)L(\theta_i,a) =m_\pi(x)\min_a\sum_iP(\theta_i\mid x)L(\theta_i,a).

This is simply the definition of the posterior: each weight \(\pi_i p_{\theta_i}(x)\) equals \(m_\pi(x)P(\theta_i\mid x)\), and the positive factor \(m_\pi(x)\) does not change which action minimizes the expression. The factorization separates the chance of seeing \(x\) from the smallest posterior loss after seeing it.

Worked Example: Squared-Error Estimation in a Binomial Model

Let \(X\mid\theta\sim\operatorname{Binomial}(2,\theta)\), with \(\Theta=\{0,\tfrac12,1\}\). Assign prior probability \(1/3\) to each parameter, use actions \(a\in\mathbb R\), and take squared-error loss \(L(\theta,a)=(a-\theta)^2\). For any posterior distribution, write \(\mu=E[\theta\mid X=x]\). Expanding the square gives

E[(a-\theta)^2\mid x] =\operatorname{Var}(\theta\mid x)+(a-\mu)^2.

The posterior loss is minimized at \(a=\mu\), with minimum \(\operatorname{Var}(\theta\mid x)\). We now calculate the predictive probabilities and these variances for all three possible observations.

For \(x=0\), the likelihoods at \(\theta=0,\tfrac12,1\) are \(1,\tfrac14,0\). The prior-predictive probability is \(m_\pi(0)=(1+\tfrac14+0)/3=5/12\). The posterior probabilities on \(0\) and \(\tfrac12\) are \(4/5\) and \(1/5\); hence its mean is \(1/10\), its second moment is \(1/20\), and its variance is \(1/20-(1/10)^2=1/25\).

For \(x=1\), the likelihoods are \(0,\tfrac12,0\). Thus \(m_\pi(1)=1/6\), the posterior puts probability one on \(\theta=\tfrac12\), and its variance is zero. For \(x=2\), the likelihoods are \(0,\tfrac14,1\), giving \(m_\pi(2)=5/12\). The posterior probabilities on \(\tfrac12\) and \(1\) are \(1/5\) and \(4/5\). Its mean is \(9/10\), its second moment is \(17/20\), and its variance is \(17/20-(9/10)^2=1/25\). Therefore,

r(\pi)=\frac{5}{12}\frac{1}{25}+\frac{1}{6}(0)+\frac{5}{12}\frac{1}{25} =\frac{1}{30}.

The posterior-minimum calculation uses the sample size \(n=2\) exactly: all three possible binomial outcomes are included, and their predictive probabilities sum to \(5/12+1/6+5/12=1\).

Concavity and Comparing Candidate Priors

Bayes risk is concave in the prior. This matters because optimizing a prior means maximizing its Bayes risk, not minimizing it. Concavity gives a useful way to handle symmetric finite models: averaging priors over parameter relabelings cannot decrease their Bayes risk when relabeling leaves the experiment and loss unchanged.

Theorem: For a fixed finite decision problem, \(r(\pi)\) is a concave function of the prior. In other words, for priors \(\pi,\rho\) and \(0\le t\le1\), \(r(t\pi+(1-t)\rho)\ge t r(\pi)+(1-t)r(\rho)\).

To prove this, fix any rule \(\delta\) and write \(g_\delta(\pi)=\sum_i\pi_iR(\theta_i,\delta)\). This is linear in \(\pi\). Since \(r(\pi)=\inf_\delta g_\delta(\pi)\), for every \(\delta\),

g_\delta(t\pi+(1-t)\rho) =tg_\delta(\pi)+(1-t)g_\delta(\rho) \ge tr(\pi)+(1-t)r(\rho).

Taking the infimum over \(\delta\) on the left preserves the inequality, proving the stated concavity. \(\square\)

Worked Example: A Symmetric Three-State Classification Model

Let \(\Theta=\{1,2,3\}\), \(\mathcal X=\{1,2,3\}\), and let \(P(X=j\mid\theta=i)=0.6\) if \(j=i\) and \(0.2\) otherwise. Use actions \(a\in\{1,2,3\}\) and zero-one loss. At observation \(j\), the weighted loss is the total prior-predictive mass at \(j\) minus the largest joint mass \(\pi_iP(X=j\mid\theta=i)\). Summing over observations gives

r(\pi)=1-\sum_{j=1}^3\max_i\{\pi_iP(X=j\mid\theta=i)\}.

Under the uniform prior, each observation has predictive probability \(1/3\). The joint mass for the matching state is \((1/3)(0.6)=0.2\); each other joint mass is \((1/3)(0.2)=1/15\). The maximum at each observation is \(0.2\), so \(r(\pi)=1-3(0.2)=0.4\).

In fact, no other prior has larger Bayes risk. Relabeling the three states and observations together leaves this model and its loss unchanged, so it leaves Bayes risk unchanged. Take any prior \(\pi\) and average its six permutations; the resulting prior is uniform. By concavity, the risk at this average is at least the average of the risks at the six permuted priors. Those six risks all equal \(r(\pi)\). Thus \(0.4=r(\text{uniform})\ge r(\pi)\), establishing that the uniform prior maximizes Bayes risk.

A Reliable Calculation and a Common Pitfall

For a finite observation model, the decomposition leads to a direct workflow:

List the possible observations.
Include every point with positive predictive probability under the candidate prior.

Compute joint weights.
For each observation \(x\) and parameter \(\theta_i\), calculate \(\pi_i p_{\theta_i}(x)\).

Minimize at each observation.
Use the specified loss to find the action with minimum weighted loss, or factor out \(m_\pi(x)\) and minimize posterior expected loss.

Sum the minima.
Add the observation-wise contributions, assigning zero contribution to any observation with zero predictive probability.

A common mistake is to choose one action that minimizes prior expected loss before observing \(X\), then use that same action for every observation. That solves a different problem: it forbids the decision rule from responding to the data. The Bayes risk instead minimizes separately at each \(x\), because a rule can assign different actions to different observations. A second error is to define a posterior at an observation with \(m_\pi(x)=0\); Bayes' rule does not define one there, and none is needed because that observation contributes zero risk under the prior.

The decomposition evaluates a prior's Bayes risk exactly in a finite model, while concavity helps organize comparisons across priors. Neither fact alone establishes that a candidate prior is least favorable for a broader minimax problem; that requires connecting its Bayes risk to the relevant risk bounds and decision rules.

Check Your Understanding

Use the definitions and calculations above to test your understanding.

Why does an observation with zero prior-predictive probability contribute zero to the weighted-minimum formula even though its posterior is undefined?
In a finite decision problem, explain why a rule that minimizes weighted loss separately at each observation attains the Bayes risk.
For squared-error loss, why is the posterior mean a minimizing action, and what is the corresponding minimum posterior risk?
What does concavity of Bayes risk imply when a prior is averaged over symmetry transformations that preserve the model and loss?
Why can choosing one fixed action before observing the data produce a risk larger than the Bayes risk over all decision rules?