Pointwise Minimization and Bayes Rules

From Conditional Decisions to Overall Risk

A decision rule uses a predictor $X$ to choose an action $g(X)$. The rule is judged by its population risk: its expected loss over the joint distribution of predictors and outcomes. Conditional risk separates that expectation into decisions made at different predictor values. The central question is whether choosing the best action separately at each value of $X$ also gives the best overall rule.

The answer is yes when a measurable rule attains the conditional minimum almost everywhere and the risks are well-defined. “Almost everywhere” matters: predictor values that occur with probability zero cannot affect population risk. The conclusion is a guarantee about a rule that already exists; it does not say that a measurable pointwise minimizer always exists.

Definition: Let $X$ be a predictor, $Y$ an outcome, $\mathcal A$ an action space, and $L(a,y)$ a nonnegative measurable loss for action $a\in\mathcal A$ when the outcome is $y$. A decision rule is a measurable function $g$ from the predictor space to $\mathcal A$. Given a conditional distribution $K(dy\mid x)$ of $Y$ given $X=x$, the conditional risk of action $a$ at $x$ is $$ r(x,a)=\int L(a,y)\,K(dy\mid x). $$ The population risk of $g$ is $R(g)=E[L(g(X),Y)]$.

For precision, suppose the predictor and outcome spaces are standard Borel spaces, so a regular conditional distribution $K(dy\mid x)$ is available. Assume the loss is jointly measurable and nonnegative. Then conditional risk is understood as a measurable function of $(x,a)$, and population risks may initially be extended nonnegative expectations. In comparisons below, we will specify when risks are finite. Conditional distributions, and hence conditional risks, are only determined up to changes on sets of predictor values having probability zero.

The Pointwise-Minimization Principle

A pointwise minimizer is a rule $g^*$ whose action minimizes $r(x,a)$ for almost every predictor value $x$. This means that, outside one set of predictor probability zero, its conditional risk is no greater than the conditional risk of any available action. The quantifier “any action” is part of the condition; selecting an action that beats only a few alternatives would not suffice.

Theorem: Suppose $g^*$ is a measurable decision rule and, for $P_X$-almost every $x$, $$ r(x,g^*(x))\leq r(x,a)\quad\text{for every }a\in\mathcal A. $$ Then $R(g^*)\leq R(h)$ for every measurable decision rule $h$. This conclusion holds for nonnegative extended risks; in particular, it applies when the risks being compared are finite.

Proof. The conditional-distribution identity for nonnegative measurable losses gives, for every measurable rule $g$, $$ R(g)=E[L(g(X),Y)]=\int r(x,g(x))\,P_X(dx). $$ To see why the action may depend on $x$ in this identity, condition on $X$: given $X=x$, the rule chooses the fixed action $g(x)$, so the conditional expectation of the loss is $\int L(g(x),y)K(dy\mid x)=r(x,g(x))$. Integrating that conditional expectation over $P_X$ gives the displayed identity. Nonnegativity permits the conditional integration even when an expectation is infinite.

Apply the pointwise assumption with $a=h(x)$. For $P_X$-almost every $x$, $r(x,g^*(x))\leq r(x,h(x))$. Integrating both nonnegative functions with respect to $P_X$ and using the identity above yields $R(g^*)\leq R(h)$. Thus $g^*$ minimizes population risk among measurable rules. $\square$

Worked Example: Choosing the Best Action at Each Predictor Value

Let $X\in\{0,1\}$, with $P(X=0)=0.6$ and $P(X=1)=0.4$, and let $Y=X$. The available actions are $u$ and $v$. Their losses are $L(u,0)=1$, $L(v,0)=4$, $L(u,1)=5$, and $L(v,1)=2$. Find the pointwise minimizing rule and compare it with the two constant rules.

Because $Y=X$, the conditional risk at each predictor value is just the loss for that outcome. At $x=0$, the conditional risks are $r(0,u)=1$ and $r(0,v)=4$, so $u$ is better. At $x=1$, they are $r(1,u)=5$ and $r(1,v)=2$, so $v$ is better. Thus the pointwise minimizing rule is $g^*(0)=u,\ g^*(1)=v$.

Its population risk is $$ R(g^*)=0.6(1)+0.4(2)=0.6+0.8=1.4. $$ The constant rule $h(x)=u$ has risk $0.6(1)+0.4(5)=0.6+2=2.6$. The constant rule $h(x)=v$ has risk $0.6(4)+0.4(2)=2.4+0.8=3.2$. Both are worse than the pointwise minimizing rule, as the theorem guarantees.

Conditional Improvement Means Lower Population Risk

The same integration argument compares any two rules; neither needs to be a pointwise minimizer. A rule is better overall if its conditional risk is no larger almost everywhere. If it improves conditional risk strictly on a set of positive predictor probability, with no compensating deterioration elsewhere, its overall risk is strictly lower.

Result: Let $g$ and $h$ be measurable rules with finite population risks. If $r(x,h(x))\leq r(x,g(x))$ for $P_X$-almost every $x$, then $R(h)\leq R(g)$. If the inequality is strict on a set of positive $P_X$-probability, then $R(h)<R(g)$.

Proof. By the conditional-risk identity, $$ R(g)-R(h)=\int \bigl(r(x,g(x))-r(x,h(x))\bigr)\,P_X(dx). $$ The integrand is nonnegative almost everywhere under the stated assumption, so its integral is nonnegative and $R(h)\leq R(g)$. If it is strictly positive on a set of positive probability, its integral is strictly positive: indeed, that set is the union over positive integers $k$ of the sets where the integrand is at least $1/k$. At least one of those sets must have positive probability, making the integral positive. Therefore $R(g)-R(h)>0$. $\square$

Worked Example: A Low-Risk Rule That Is Not Conditionally Optimal

Let $X\in\{0,1\}$, with $P(X=0)=0.99$ and $P(X=1)=0.01$, and let $Y=X$. There are two actions, $A$ and $B$, with losses $L(A,0)=0,\ L(B,0)=1,\ L(A,1)=4,\ L(B,1)=0$. Compare the constant rule $g(x)=A$ with the pointwise minimizing rule.

At $x=0$, the conditional risks are $r(0,A)=0$ and $r(0,B)=1$, so $A$ minimizes risk. At $x=1$, they are $r(1,A)=4$ and $r(1,B)=0$, so $B$ minimizes risk. The pointwise minimizing rule takes $A$ at 0 and $B$ at 1, and its population risk is $0.99(0)+0.01(0)=0$.

The constant rule $g(x)=A$ is not conditionally optimal at $x=1$: its conditional risk there is 4, while action $B$ has conditional risk 0. Nevertheless, its population risk is $0.99(0)+0.01(4)=0.04$, which may look small on the scale of the losses. Low overall risk does not imply conditional optimality. Here the rare predictor value has little weight in the population average, but it still has a strictly better action.

Almost-Sure Qualification

The pointwise principle is a statement for predictor values that occur under $P_X$, not necessarily for every value in the predictor space. Conditional distributions can be changed on a set of predictor probability zero without changing the joint distribution of $(X,Y)$ or the population risk of any rule. Consequently, a claimed action at an impossible predictor value has no bearing on global optimality.

Worked Example: A Conditional Risk at an Impossible Predictor Value

Suppose $X$ takes values 0 and 1 with probability $1/2$ each; the value 2 is never observed. Let $Y=X$, and let actions be $L$ and $R$, with losses $L(L,0)=0,\ L(R,0)=2,\ L(L,1)=3,\ L(R,1)=0$.

At $x=0$, the conditional risks are 0 for $L$ and 2 for $R$, so $L$ is best. At $x=1$, they are 3 for $L$ and 0 for $R$, so $R$ is best. The rule choosing $L$ at 0 and $R$ at 1 has risk 0.

There is no uniquely determined conditional distribution of $Y$ given $X=2$: since $P(X=2)=0$, the conditional distribution at that value can be assigned arbitrarily without changing the joint distribution. One version could make $L$ appear best at 2; another could make $R$ appear best. Either choice gives the same population risk, because the event $X=2$ has probability zero. The meaningful pointwise claim is therefore almost sure, not necessarily a claim at every imaginable predictor value.

What the Principle Does Not Guarantee

The theorem is a sufficient route from conditional optimality to global optimality: once a measurable pointwise minimizing rule is available, it is globally risk-minimizing. It does not, by itself, construct that rule. In particular, choosing a minimizing action separately at every $x$ is not enough unless those choices form a measurable function of $x$. The conditions that ensure such a measurable choice are a separate issue.

Nor should the theorem be reversed casually. A comparison of a few candidate rules, or a small value of $R(g)$, does not establish that $g(x)$ minimizes $r(x,a)$ at each predictor value. A global comparison gives an average over $P_X$; poor conditional performance in a rare region may contribute little to that average. To establish conditional optimality, compare conditional risks at the relevant $x$, rather than relying on the overall risk alone.

Common Pitfall

Wrong reasoning: “This rule has very low population risk, so its action must be best whenever that predictor value occurs.” The conclusion does not follow. Population risk averages conditional risks with weights given by the predictor distribution. A poor action in a rare region can make only a small contribution to that average, as the preceding example showed.

Correction: To prove a rule is globally optimal by pointwise minimization, verify that it is measurable and that its conditional risk is no greater than that of every action for $P_X$-almost every $x$. Then integrate the conditional risks. To show only that a rule has low risk, report its population risk without treating that fact as evidence of conditional optimality.

Key takeaway: Conditional risk is averaged over predictor values to produce population risk. A measurable rule that minimizes conditional risk almost everywhere therefore minimizes population risk, but a rule with low population risk need not minimize conditional risk at each predictor value.

Check Your Understanding

Use the conditional-risk comparison and its assumptions to answer each question.

What does it mean for a measurable rule $g^*$ to minimize conditional risk almost everywhere?
Why does integrating a pointwise conditional-risk inequality imply an inequality between population risks?
A rule has low population risk but uses a conditionally inferior action on a rare set of predictor values. Is that possible? Explain.
Why can conditional risk at a predictor value with probability zero depend on the chosen version of the conditional distribution?
Does choosing a minimizing action separately at every $x$ automatically produce a valid decision rule? What additional property is needed?