Existence and Uniqueness of Bayes Rules

Why Existence and Uniqueness Need Separate Conditions

A Bayes rule chooses an action to minimize conditional risk at each observed value of the covariates. Two questions are hidden in that description: does a minimizing action actually exist, and, if it does, is there only one? These are different issues. A risk can have a minimum without having a unique minimizer, and it can have an infimum that no action attains.

The existence question is first a pointwise optimization problem. Compactness of the action space and lower semicontinuity of the conditional risk give a minimizer at each covariate value. To obtain a Bayes rule, however, those pointwise choices must also be made measurably. The measurable-selection result for Bayes decisions supplies this step under standard measurability conditions. Strict convexity gives the separate conclusion of uniqueness, provided the action space is convex.

Definitions and Conditions

Definition: Let $X$ denote the covariates, let $\mathcal{A}$ be the action space, and let $r(x,a)$ be the conditional risk of taking action $a$ when $X=x$. The set of conditional-risk minimizers is $$ \Gamma(x)=\operatorname*{arg\,min}_{a\in\mathcal{A}} r(x,a). $$ A Bayes rule is a measurable function $g$ from covariate values to actions that minimizes population risk among measurable decision rules. In the conditional-risk formulation, this means $g(x)\in\Gamma(x)$ for almost every covariate value.

Here and below, “almost every” refers to the distribution of $X$. Thus, changing a decision rule on a set of covariate values with probability zero does not change its status as a Bayes rule. Uniqueness of a Bayes rule will mean equality almost surely, not equality at every possible value of $x$.

For the existence result, assume that the covariate space is a standard Borel space, the action space $\mathcal{A}$ is a nonempty compact metric space, and $r(x,a)$ is jointly measurable in $(x,a)$ and lower semicontinuous in $a$ for each $x$. Assume also that the risk is finite and that there is a fixed action $a_0\in\mathcal{A}$ with $E[r(X,a_0)]<\infty$. Conditional risks are typically nonnegative; that assumption ensures the expectations used below are well defined. In statistical applications, $r(x,a)$ is a version of $E[L(a,Y)\mid X=x]$, where $L$ is the loss.

Definition: A function $a\mapsto r(x,a)$ is lower semicontinuous if, whenever $a_k\to a$, it satisfies $$ r(x,a)\leq \liminf_{k\to\infty}r(x,a_k). $$ It is strictly convex on a convex action space if, for all distinct $a,b\in\mathcal{A}$ and all $t\in(0,1)$, $$ r(x,ta+(1-t)b)<t\,r(x,a)+(1-t)\,r(x,b). $$

Lower semicontinuity rules out a downward “hole” at a point that would make nearby risks approach a value lower than the risk at that point. It does not require differentiability or continuity. Strict convexity, by contrast, concerns the shape of the risk between actions and will be used to establish uniqueness.

Worked Examples

Worked Example: A Compact Action Space Gives an Attained Minimum

Suppose the action space is $\mathcal{A}=[0,1]$, and at a particular covariate value the conditional risk is $r(x,a)=(a-1.4)^2$. Find its minimizer on the allowed action space.

The unconstrained target $1.4$ is outside $[0,1]$, so it is not an admissible action. To check the risk over the allowed interval, compare it with the risk at $a=1$: $$ r(x,a)-r(x,1)=(a-1.4)^2-(1-1.4)^2 =a^2-2.8a+1.8=(a-1)(a-1.8). $$ For $0\leq a<1$, both $a-1$ and $a-1.8$ are negative, so their product is positive. At $a=1$, the difference is zero. Therefore $a=1$ is the unique minimizer, and its risk is $r(x,1)=0.16$. This example also shows why the minimizer need not be an unconstrained stationary point: the optimum can lie on the boundary.

Worked Example: Strict Convexity Produces a Unique Minimizer

Let $\mathcal{A}=[0,1]$ and $r(x,a)=(a-0.3)^2+0.21$ for a fixed $x$. Verify the minimizer and its uniqueness.

For every $a\in[0,1]$, the squared term is nonnegative, so $r(x,a)\geq 0.21$. Equality holds exactly when $a=0.3$, which is an allowed action. Thus $a=0.3$ is the unique minimizer and the minimum risk is $0.21$.

The risk is strictly convex: for distinct $a,b$ and $t\in(0,1)$, direct expansion gives $$ t(a-0.3)^2+(1-t)(b-0.3)^2-\bigl(ta+(1-t)b-0.3\bigr)^2 =t(1-t)(a-b)^2>0. $$ The constant $0.21$ does not affect this inequality. This calculation illustrates the strict-convexity condition used in the uniqueness result.

Worked Example: A Pointwise Choice Becomes a Measurable Rule

Suppose $X$ takes the two values $A$ and $B$, with $P(X=A)=0.4$ and $P(X=B)=0.6$. Let $\mathcal{A}=[0,1]$, with conditional risks $r(A,a)=(a-0.2)^2$ and $r(B,a)=(a-0.8)^2$. Find a Bayes rule and compare it with the constant rule $g(x)=0.2$.

At $X=A$, the risk is minimized at $a=0.2$; at $X=B$, it is minimized at $a=0.8$. The rule $g(A)=0.2,\ g(B)=0.8$ is measurable because every function on this finite covariate space is measurable. Its population risk is $$ 0.4(0.2-0.2)^2+0.6(0.8-0.8)^2=0. $$ The constant rule $g(x)=0.2$ instead has risk $$ 0.4(0.2-0.2)^2+0.6(0.2-0.8)^2 =0+0.6(0.36)=0.216. $$ The pointwise choices therefore form a Bayes rule, and the constant rule is not optimal.

Existence of a Bayes Rule

Theorem: Suppose $\mathcal{A}$ is a nonempty compact metric space, the covariate space is standard Borel, and $r(x,a)$ is jointly measurable, finite, and lower semicontinuous in $a$ for each $x$. Then $\Gamma(x)$ is nonempty for every $x$, and there is a measurable rule $g$ with $g(x)\in\Gamma(x)$. If $r$ is a nonnegative conditional risk and $E[r(X,a_0)]<\infty$ for some fixed $a_0\in\mathcal{A}$, this rule has finite population risk and is a Bayes rule.

Proof. Fix $x$. Since $r(x,\cdot)$ is finite and lower semicontinuous on a nonempty compact space, it attains its minimum. One way to see this is to take a sequence of actions whose risks converge to the infimum. Compactness gives a convergent subsequence with limit $a^*\in\mathcal{A}$, and lower semicontinuity implies that the risk at $a^*$ is no greater than the infimum. Hence $a^*\in\Gamma(x)$, so $\Gamma(x)$ is nonempty.

The measurable-selection theorem for measurable functions with compact action space and lower-semicontinuous sections gives a measurable choice $g(x)\in\Gamma(x)$. This is where measurability matters: pointwise existence alone does not establish that the choices can be assembled into a permissible decision rule.

For each $x$, minimization gives $r(x,g(x))\leq r(x,a_0)$. Nonnegativity and the assumed integrability of $r(X,a_0)$ therefore imply $$ 0\leq E[r(X,g(X))]\leq E[r(X,a_0)]<\infty. $$ For any other measurable rule $h$, pointwise minimization gives $r(x,g(x))\leq r(x,h(x))$. Taking expectations shows that $g$ minimizes population risk. Thus $g$ is a Bayes rule. $\square$

Compactness and lower semicontinuity are sufficient conditions for attainment, not a claim that every problem lacking either condition has no minimizer. They provide a reliable guarantee without requiring a special analysis of each risk function.

Strict Convexity and Uniqueness

Theorem: Suppose $\mathcal{A}$ is convex and $r(x,\cdot)$ is strictly convex on $\mathcal{A}$ for almost every $x$. Whenever a conditional-risk minimizer exists, it is unique for almost every $x$. If a Bayes rule exists under these conditions, it is unique up to equality almost surely.

Proof. Fix an $x$ for which the conditional risk is strictly convex, and suppose two distinct actions $a$ and $b$ both minimize it. Convexity of $\mathcal{A}$ means their midpoint $(a+b)/2$ is an allowed action. Strict convexity then gives $$ r\left(x,\frac{a+b}{2}\right)<\frac{r(x,a)+r(x,b)}{2}. $$ Since both actions have the minimum risk, the right-hand side equals that minimum. The midpoint would have strictly smaller risk, a contradiction. Therefore the conditional minimizer is unique for almost every $x$.

Now let $g$ and $h$ both be Bayes rules. Each is a conditional-risk minimizer almost surely, so at every $x$ outside the union of the relevant null sets, both must equal the unique element of $\Gamma(x)$. Hence $g(X)=h(X)$ almost surely. This does not require the two rule functions to agree at covariate values that occur with probability zero. $\square$

The two theorems answer different questions. Compactness and lower semicontinuity help establish that a minimizer is attained; strict convexity rules out two different minimizers. Strict convexity is not needed for existence, and the existence of a minimizer does not imply strict convexity.

When the Conditions Fail

Worked Example: An Infimum That Is Not a Minimum

First take the noncompact action space $\mathcal{A}=(0,1)$ and the risk $r(a)=a$. Its infimum is $0$, since actions can be chosen arbitrarily close to $0$, but there is no action $a\in(0,1)$ with $r(a)=0$. Thus the infimum is not attained.

Now keep the compact action space $[0,1]$, but define $r(0)=1$ and $r(a)=a$ for $a>0$. The infimum is again $0$, approached as $a$ decreases to $0$, and no action attains it: the risk is $1$ at $0$ and positive at every $a>0$. This function is not lower semicontinuous at $0$, because $$ r(0)=1>\liminf_{a\to 0,\ a>0}r(a)=0. $$ The examples show how either a missing boundary point or a failure of lower semicontinuity can defeat the attainment guarantee.

Uniqueness can fail even when existence is secure. For example, on the compact convex action space $[0,1]$, the constant risk $r(a)=2$ is continuous and every action minimizes it. The risk is not strictly convex. Strict convexity is a sufficient condition for uniqueness, rather than a necessary one.

Common Mistake: Confusing a Unique Action with a Unique Rule Everywhere

A common mistake is to argue that strict convexity makes two Bayes rules identical at every possible covariate value. Strict convexity gives a unique conditional minimizer where it holds, but Bayes optimality is an almost-sure property. If $X$ never takes a particular value, changing a rule only at that value does not change its population risk. The correction is to state uniqueness as equality almost surely, unless additional conditions ensure that all covariate values of interest have positive probability or otherwise require pointwise agreement.

Key takeaway: Compactness and lower semicontinuity ensure that conditional minima are attained; measurability allows pointwise minimizers to form a decision rule. Strict convexity on a convex action space makes the minimizer unique, and the resulting Bayes rule is unique up to probability-zero differences.

Check Your Understanding

Use the conditions and distinctions developed above to answer these questions.

Why does pointwise existence of a minimizer not, by itself, guarantee a Bayes rule?
Which part of the existence argument uses compactness, and which part uses lower semicontinuity?
Can a conditional risk have a minimum without being strictly convex? Give an example.
Why must the action space be convex for the strict-convexity uniqueness argument?
What does uniqueness of a Bayes rule mean when the covariate distribution assigns probability zero to some values?