Finite-Outcome Bayes-Rule Calculation Workshop

From Posterior Probabilities to an Action

A conditional Bayes-rule calculation starts with a distribution for the outcome given the information available, then compares the expected loss of each permissible action. The arithmetic is often simple; the important discipline is to use the same posterior probabilities and the specified loss for every candidate action. The action with the most favorable outcome in mind is not necessarily the action with the smallest conditional risk.

Let $Y$ take values in the finite set $\{y_1,\ldots,y_m\}$, and let $\mathcal A=\{a_1,\ldots,a_k\}$ be a finite action set. At a fixed information value $x$, write $p_j(x)=P(Y=y_j\mid X=x)$. These probabilities are nonnegative and sum to one. For loss $L(a,y)$, the conditional risk of action $a$ is the posterior-weighted average of its losses across possible outcomes.

Definition: The conditional risk of action $a\in\mathcal A$, at information value $x$, is $$ R(a\mid x)=E[L(a,Y)\mid X=x] =\sum_{j=1}^{m}p_j(x)L(a,y_j). $$ A conditional Bayes action is any action in $\mathcal A$ minimizing $R(a\mid x)$. If several actions attain the minimum, all are conditional Bayes actions.

This definition is also a practical recipe: list the posterior probabilities, calculate each action’s loss for each outcome, multiply and sum, then compare the resulting risks. The calculation is conditional on $x$; probabilities averaged over different information values generally cannot replace the posterior probabilities at the $x$ being considered.

Why Minimizing Conditional Risk Gives the Bayes Rule

The finite action set makes the pointwise optimization especially direct. For each $x$, calculate the risk of every action and select a minimizer. If $X$ is random and the losses are integrable, this pointwise rule also minimizes expected loss over measurable decision rules.

Theorem: Suppose $\mathcal A$ is finite, $Y$ has finitely many possible values, the losses $L(a,Y)$ are integrable, and the conditional probabilities $P(Y=y_j\mid X)$ are measurable. Then any measurable rule $\delta(X)$ that minimizes $R(a\mid X)$ over $a\in\mathcal A$ almost surely minimizes $E[L(\delta(X),Y)]$ among measurable $\mathcal A$-valued rules.

Proof. For any measurable rule $\delta$, the defining property of conditional expectation gives $E[L(\delta(X),Y)]=E[R(\delta(X)\mid X)]$. This follows by writing the loss as a finite sum over actions and outcomes, or by conditioning on $X$. By pointwise minimization, for any other measurable rule $\gamma$, $R(\delta(X)\mid X)\leq R(\gamma(X)\mid X)$ almost surely. Taking expectations preserves the inequality, so $E[L(\delta(X),Y)]\leq E[L(\gamma(X),Y)]$. A measurable minimizer exists: order the finite actions in advance and choose the first action attaining the minimum. Its selection sets are defined by finitely many comparisons between measurable risks, so they are measurable. $\square$

The theorem justifies the calculation at each information value. It does not say that the action with the smallest risk at one particular $x$ must be best at every other $x$; the posterior probabilities, and hence the risk comparisons, can change with $x$.

Worked Examples Under Four Losses

Worked Example: Zero-One Loss With Three Outcomes

Suppose the possible labels are $A,B,C$, with posterior probabilities $0.46, 0.34, 0.20$, respectively. The action is a predicted label, and zero-one loss is $L(a,y)=1$ when $a\neq y$ and $0$ when $a=y$.

For action $A$, the loss is zero when the outcome is $A$, and one otherwise: $R(A\mid x)=0.46(0)+0.34(1)+0.20(1)=0.54$.

For the other labels, $R(B\mid x)=0.46(1)+0.34(0)+0.20(1)=0.66$, and $R(C\mid x)=0.46(1)+0.34(1)+0.20(0)=0.80$. The minimum is $0.54$, so $A$ is the conditional Bayes action. Here the calculation agrees with selecting the label having the largest posterior probability.

Result: Under zero-one loss for a finite label set, $R(a\mid x)=1-P(Y=a\mid X=x)$. Thus, the conditional Bayes actions are exactly the labels with maximal posterior probability.

Derivation. If the action is $a$, its loss is zero on the event $Y=a$ and one on its complement. Therefore $R(a\mid x)=0\cdot P(Y=a\mid x)+1\cdot P(Y\neq a\mid x)=1-P(Y=a\mid x)$. Minimizing this quantity is equivalent to maximizing $P(Y=a\mid x)$. If the largest posterior probability is shared, each label attaining it has the same minimum risk. $\square$

Worked Example: Asymmetric Costs in a Binary Decision

Let $Y=1$ mean that a condition is present and $Y=0$ mean that it is absent. At the information value under consideration, suppose $P(Y=1\mid x)=0.30$ and $P(Y=0\mid x)=0.70$. A positive decision when the condition is absent costs 2; a negative decision when it is present costs 5. Correct decisions cost 0.

For positive action $a=1$, the only nonzero loss occurs when $Y=0$: $R(1\mid x)=0.70(2)+0.30(0)=1.40$. For negative action $a=0$, the nonzero loss occurs when $Y=1$: $R(0\mid x)=0.30(5)+0.70(0)=1.50$. Because $1.40<1.50$, the positive action minimizes conditional risk, even though the condition is less likely than not.

More generally, if the false-positive cost is $c_{\mathrm{FP}}>0$, the false-negative cost is $c_{\mathrm{FN}}>0$, and $p=P(Y=1\mid x)$, then $R(1\mid x)=c_{\mathrm{FP}}(1-p)$ and $R(0\mid x)=c_{\mathrm{FN}}p$. Choosing 1 is at least as good as choosing 0 exactly when $c_{\mathrm{FP}}(1-p)\leq c_{\mathrm{FN}}p$, or $p\geq c_{\mathrm{FP}}/(c_{\mathrm{FP}}+c_{\mathrm{FN}})$. For the stated costs, this threshold is $2/7$, and $0.30>2/7$, confirming the calculation.

Worked Example: Squared Error With a Finite Action Set

Suppose $Y$ takes values $0,2,5$ with probabilities $0.20,0.50,0.30$, and the only allowed actions are $\mathcal A=\{0,2,4\}$. Under squared-error loss $L(a,y)=(a-y)^2$, calculate the risk of each feasible action.

For action 0, $R(0\mid x)=0.20(0-0)^2+0.50(0-2)^2+0.30(0-5)^2 =0+2+7.5=9.5$. For action 2, $R(2\mid x)=0.20(2-0)^2+0.50(2-2)^2+0.30(2-5)^2 =0.8+0+2.7=3.5$. For action 4, $R(4\mid x)=0.20(4-0)^2+0.50(4-2)^2+0.30(4-5)^2 =3.2+2+0.3=5.5$. The unique minimum is 3.5, so the conditional Bayes action within this finite action set is 2.

The posterior mean is $E[Y\mid x]=0.20(0)+0.50(2)+0.30(5)=2.5$. The result can be checked using the squared-error identity $R(a\mid x)=\operatorname{Var}(Y\mid x)+(a-E[Y\mid x])^2$. Here $E[Y^2\mid x]=0.50(4)+0.30(25)=9.5$, so the variance is $9.5-(2.5)^2=3.25$. The risks at actions 0, 2, and 4 are therefore $3.25+2.5^2=9.5$, $3.25+0.5^2=3.5$, and $3.25+1.5^2=5.5$, as calculated directly.

Result: For a finite posterior distribution with finite second moment, let $\mu=E[Y\mid x]$. Under squared-error loss, $$ R(a\mid x)=\operatorname{Var}(Y\mid x)+(a-\mu)^2. $$ Consequently, among a finite feasible action set, the minimizers are the feasible actions closest to $\mu$; there can be a tie.

Proof. Expand the square and use $E[Y-\mu\mid x]=0$: $E[(a-Y)^2\mid x]=E[((a-\mu)-(Y-\mu))^2\mid x]$ $=(a-\mu)^2-2(a-\mu)E[Y-\mu\mid x]+E[(Y-\mu)^2\mid x]$ $=(a-\mu)^2+\operatorname{Var}(Y\mid x)$. The variance term does not depend on $a$, so minimizing risk over the feasible actions is equivalent to minimizing squared distance from $\mu$. $\square$

Worked Example: Absolute Error and a Restricted Action Set

Let $Y$ take values $0,1,4$ with probabilities $0.20,0.50,0.30$, and suppose the feasible actions are $\{0,2,4\}$. Under absolute-error loss $L(a,y)=|a-y|$, the risks are $R(0\mid x)=0.20(0)+0.50(1)+0.30(4)=1.70$, $R(2\mid x)=0.20(2)+0.50(1)+0.30(2)=1.50$, and $R(4\mid x)=0.20(4)+0.50(3)+0.30(0)=2.30$. Thus action 2 is best among the feasible actions.

The posterior median is 1: the cumulative probability at 1 is $0.20+0.50=0.70$, while the probability strictly below 1 is $0.20$. Since 1 is not feasible, the unrestricted median cannot be selected. The direct comparison shows that 2 is the best available action; it is important not to assume that a restricted action set contains a median.

For finite outcome distributions, the median connection can be checked by looking at how absolute-error risk changes as the action moves. Write $F(a)=P(Y\leq a\mid x)$ and $F(a-)=P(Y<a\mid x)$. The right-hand slope of $R(a\mid x)$ is $2F(a)-1$: outcomes at or below $a$ contribute slope $+1$, and outcomes above $a$ contribute $-1$. The left-hand slope is $2F(a-)-1$. Since this risk is piecewise linear and convex, an unrestricted minimizer occurs where the left-hand slope is nonpositive and the right-hand slope is nonnegative. Equivalently, $F(a-)\leq 1/2\leq F(a)$, the usual median condition. With a restricted finite action set, compare risks at its feasible actions unless it contains an unrestricted median.

Common Pitfalls and a Reliable Calculation

A frequent mistake is to choose an action by a familiar shortcut without checking whether its assumptions match the problem. The largest posterior probability gives a zero-one Bayes action, but it need not minimize an asymmetric cost matrix. The posterior mean minimizes squared-error risk over unrestricted real actions, but when actions are restricted to a finite set, compare feasible actions to the mean. A median minimizes unrestricted absolute-error risk, but a finite action set may not contain one.

Another mistake is to omit outcomes where the action has zero loss, or to use the same cost for both kinds of error when the specified costs differ. Write one risk expression per action before comparing them. For finite posterior distributions, each expression is a finite weighted sum, so the calculation can be checked term by term.

Fix the information value.
Record the posterior probability of every possible outcome at the $x$ being analyzed.

List feasible actions and losses.
Use the stated action set and loss function; do not silently add actions or change costs.

Calculate every conditional risk.
Multiply each outcome’s loss by its posterior probability and add across outcomes.

Compare and report ties.
Select every action attaining the minimum, and state the minimum risk when it helps verify the comparison.

Key takeaway: A Bayes action is determined by posterior expected loss, not by posterior probability alone. The loss function and the feasible action set are both part of the decision problem.

Check Your Understanding

For each question, identify the relevant risk comparison before choosing an action.

Under zero-one loss, posterior probabilities for three labels are $0.25,0.25,0.50$. What is the minimum conditional risk, and which action attains it?
In a binary problem, the false-positive cost is 3 and the false-negative cost is 1. Derive the posterior-probability threshold for choosing the positive action.
For squared-error loss, why can the posterior mean fail to be an allowed action when the action set is finite?
If an unrestricted posterior median is absent from the feasible action set, what calculation determines the best feasible action under absolute-error loss?
Explain why pointwise minimization of conditional risk minimizes expected loss when the action set is finite.