Best-in-Class Risk Under Misspecification

Why the Best Available Predictor May Not Be Best Overall

A predictor is judged by how well it performs on the population from which future observations are drawn. But statistical learning usually restricts the available predictors: a model may allow only constant classifiers, linear decision boundaries, or rules of a particular form. The best predictor permitted by that model can therefore have higher risk than the best predictor that could be chosen without the restriction.

This distinction matters when interpreting a fitted model. A learning procedure might successfully find the predictor with the lowest population risk in its class, yet that predictor can still make errors that no improvement in fitting, computation, or sample size will remove. Those errors arise because the class cannot represent a population-optimal predictor. The goal here is to define the two risks precisely and understand what their difference does—and does not—tell us.

Definition: Let $Z=(X,Y)$ have the population distribution, let $\mathcal{G}$ be a collection of measurable predictors $g$, and let $L(g(X),Y)$ be a nonnegative loss. The population risk of $g$ is $$ R(g)=E[L(g(X),Y)], $$ provided this expectation is defined. The best-in-class risk is $$ R_{\mathcal{G}}^*=\inf_{g\in\mathcal{G}}R(g). $$ If some $g_{\mathcal{G}}^*\in\mathcal{G}$ satisfies $R(g_{\mathcal{G}}^*)=R_{\mathcal{G}}^*$, it is a best-in-class predictor (or population-risk minimizer in $\mathcal{G}$).

An infimum is not automatically attained. Thus $R_{\mathcal{G}}^*$ is well-defined as a lower bound even when there is no predictor in $\mathcal{G}$ that achieves it. This is one reason to distinguish the best-in-class risk from a best-in-class predictor.

The unrestricted benchmark is defined by allowing all measurable predictors with well-defined risk: $$ R^*=\inf_{g\in\mathcal{A}}R(g), $$ where $\mathcal{A}$ denotes that unrestricted collection. When the infimum is attained, a minimizer is called a Bayes predictor, and $R^*$ is the Bayes risk. The benchmark depends on the loss and the population distribution, not on a particular fitted model.

Comparing the Two Risks

A restricted class is a subset of the unrestricted collection. Its best-in-class risk therefore cannot be lower than the unrestricted risk. The difference $R_{\mathcal{G}}^*-R^*$ is a population-level measure of how much worse the class's best achievable risk is than the unrestricted benchmark. It is useful to keep the risk difference separate from the question of whether a specific learning algorithm can find a good member of the class.

Proposition: If $\mathcal{G}\subseteq\mathcal{A}$, then $R_{\mathcal{G}}^*\geq R^*$. If a Bayes predictor $g^*$ belongs to $\mathcal{G}$, then $R_{\mathcal{G}}^*=R^*$, and $g^*$ is also a best-in-class predictor.

Proof. Since $\mathcal{G}\subseteq\mathcal{A}$, every risk value in the collection $\{R(g):g\in\mathcal{G}\}$ also belongs to the collection $\{R(g):g\in\mathcal{A}\}$. The infimum over the larger collection cannot exceed the infimum over the smaller collection. Hence $R^*\leq R_{\mathcal{G}}^*$. If $g^*\in\mathcal{G}$ and $R(g^*)=R^*$, then $R_{\mathcal{G}}^*\leq R(g^*)=R^*$. Together with the first inequality, this gives $R_{\mathcal{G}}^*=R^*$. $\square$

In this tutorial, a class is misspecified relative to the chosen target if it does not contain a Bayes predictor. That statement concerns representation: the target is absent from the class. A stronger statement is that the class has a strictly positive risk gap, $R_{\mathcal{G}}^*>R^*$. These statements need not be equivalent. A class can omit every exact Bayes predictor while containing predictors whose risks get arbitrarily close to the Bayes risk.

Bayes Risk for Binary Classification

For binary classification with $Y\in\{0,1\}$ and zero-one loss $L(g(X),Y)=\mathbf{1}\{g(X)\ne Y\}$, define $\eta(X)=P(Y=1\mid X)$. The conditional probability $\eta(X)$ can be defined as the conditional expectation $E[\mathbf{1}\{Y=1\}\mid X]$. At a feature value with conditional probability $\eta$, predicting 0 has conditional error $\eta$, while predicting 1 has conditional error $1-\eta$. The unrestricted best decision is therefore to choose whichever of these two errors is smaller.

Theorem: For binary classification under zero-one loss, a Bayes predictor is $g^*(X)=\mathbf{1}\{\eta(X)\geq 1/2\}$, with either label allowed when $\eta(X)=1/2$. Its risk is $R^*=E[\min\{\eta(X),1-\eta(X)\}]$. For any measurable classifier $g$, $$ R(g)-R^*=E\!\left[|2\eta(X)-1|\mathbf{1}\{g(X)\ne g^*(X)\}\right]. $$

Proof. Conditional on $X$, the zero-one loss has conditional expectation $$ E[\mathbf{1}\{g(X)\ne Y\}\mid X] =\eta(X)\mathbf{1}\{g(X)=0\}+(1-\eta(X))\mathbf{1}\{g(X)=1\}. $$ For each value of $X$, the right-hand side is either $\eta(X)$ or $1-\eta(X)$. Its minimum is $\min\{\eta(X),1-\eta(X)\}$, attained by choosing label 1 when $\eta(X)\geq 1/2$ and label 0 when $\eta(X)<1/2$. Taking expectations proves the expression for $R^*$.

If $g(X)=g^*(X)$, the two classifiers have equal conditional risk, so their conditional risk difference is zero. If they disagree and $\eta(X)>1/2$, then $g^*(X)=1$, and the conditional risk difference is $\eta(X)-(1-\eta(X))=2\eta(X)-1$. If they disagree and $\eta(X)<1/2$, then $g^*(X)=0$, and the difference is $(1-\eta(X))-\eta(X)=1-2\eta(X)$. At $\eta(X)=1/2$, the difference is zero regardless of the tie choice. Thus, in every case, the conditional excess risk is $|2\eta(X)-1|\mathbf{1}\{g(X)\ne g^*(X)\}$. Taking expectations proves the identity. $\square$

The identity makes the restricted-class comparison concrete: a classifier in $\mathcal{G}$ pays risk for disagreeing with the Bayes decision, with greater cost where the conditional class probability is farther from one half. A disagreement where $\eta(X)=1/2$ has no risk cost, because neither label is better there. Consequently, not containing a particular chosen Bayes predictor does not by itself imply a positive risk gap.

Worked Example: Constant Classifiers When the Bayes Rule Varies

Let $X$ take values $a$ and $b$, each with probability $1/2$. Suppose $P(Y=1\mid X=a)=0.9$ and $P(Y=1\mid X=b)=0.3$. Compare unrestricted classification with the class $\mathcal{G}$ containing only the two constant classifiers, $g_0(x)=0$ and $g_1(x)=1$.

At $a$, the Bayes decision is 1 and its conditional error is $1-0.9=0.1$. At $b$, the Bayes decision is 0 and its conditional error is $0.3$. Therefore $$ R^*=\tfrac12(0.1)+\tfrac12(0.3)=0.2. $$

The constant classifier $g_0$ has error $0.9$ at $a$ and $0.3$ at $b$, giving $$ R(g_0)=\tfrac12(0.9)+\tfrac12(0.3)=0.6. $$ The constant classifier $g_1$ has error $0.1$ at $a$ and $0.7$ at $b$, giving $$ R(g_1)=\tfrac12(0.1)+\tfrac12(0.7)=0.4. $$ Thus $g_1$ is the best-in-class predictor, $R_{\mathcal{G}}^*=0.4$, and the class's risk is $0.4-0.2=0.2$ above Bayes risk. The constant class cannot adapt its prediction to the two different feature values.

Restricted Rules Can Miss a Pattern

A class can be unable to represent the Bayes rule even when it contains several nonconstant predictors. For instance, a class of threshold rules can fail when the labels alternate across ordered feature values. The key comparison is still made at the population level: calculate each allowed rule's risk, then compare the smallest such risk with the unrestricted benchmark.

Worked Example: Threshold Classifiers on an Alternating Pattern

Let $X$ be uniform on $\{1,2,3,4\}$, and let $Y$ be deterministic with labels $0,1,0,1$, respectively. The Bayes predictor has risk zero because it can assign the correct label at every feature value. Restrict attention to threshold classifiers $g_t(x)=\mathbf{1}\{x\geq t\}$, with real $t$.

If $t\leq 1$, the rule predicts 1 at all four points and makes two errors, so its risk is $2/4$. If $1<t\leq 2$, its predictions are $0,1,1,1$, making one error at $x=3$, so its risk is $1/4$. If $2<t\leq 3$, its predictions are $0,0,1,1$, making errors at $x=2$ and $x=3$, so its risk is $2/4$. If $3<t\leq 4$, its predictions are $0,0,0,1$, making one error at $x=2$, so its risk is $1/4$. Finally, if $t>4$, it predicts 0 everywhere and makes two errors, so its risk is $2/4$.

The smallest risk among these cases is $R_{\mathcal{G}}^*=1/4$. The risk gap is $1/4-0=1/4$. Even a population-optimal choice of threshold cannot capture the alternating pattern.

Exclusion Does Not Always Mean a Positive Gap

A different issue arises when the restricted class contains rules that approach the Bayes risk ever more closely, but no rule in the class achieves it. This is why “the target is not in the class” and “the best-in-class risk is strictly worse” must be kept distinct. The next example computes both the infimum and whether a minimizer exists.

Worked Example: Zero Risk Gap Without an Attained Minimum

Let $X$ be uniform on $[0,1]$, let $Y=0$ with probability one, and use zero-one loss. The unrestricted Bayes predictor is the constant-zero classifier and has risk $R^*=0$. Consider the restricted class $\mathcal{G}=\{g_t:t\in(0,1)\}$, where $g_t(x)=\mathbf{1}\{x\geq t\}$. Every allowed rule predicts 1 on a set of probability $1-t$, so $$ R(g_t)=P(X\geq t)=1-t. $$ For every $t\in(0,1)$, $R(g_t)>0$, so no member has the Bayes risk. Yet for any $\varepsilon>0$, choose $t=1-\min\{\varepsilon/2,1/2\}$. This value lies in $(0,1)$ and satisfies $R(g_t)=\min\{\varepsilon/2,1/2\}<\varepsilon$. Hence $\inf_{t\in(0,1)}R(g_t)=0=R^*$, but the infimum is not attained. The Bayes predictor is excluded while the risk gap is zero.

What the Best-in-Class Risk Tells Us

The best-in-class risk is a population benchmark for a specified class and loss. It answers: if every predictor in this class were evaluated on the population, what is the lowest risk available? It does not, by itself, describe a fitted predictor or guarantee that an optimization procedure finds a minimizer. Nor does it say whether the minimizer is unique.

These distinctions are useful when describing model limitations. A positive difference $R_{\mathcal{G}}^*-R^*$ establishes that even the best risk in the class is worse than the unrestricted benchmark. If the difference is zero, however, it does not follow that the class contains a Bayes predictor: the infimum may be approached without being attained, as in the threshold example. Conversely, if a Bayes predictor belongs to the class, the risks are equal, though other predictors in the class might also attain the same risk.

A common mistake is to say, “The model cannot represent the Bayes rule, so its best possible risk must be strictly above Bayes risk.” That conclusion does not follow without additional conditions. An omitted predictor can be approximated arbitrarily well in risk by members of the class. The correct statements are separate: representation asks whether a Bayes predictor belongs to the class; a risk comparison asks whether the infimum of risks in the class is strictly larger than the Bayes risk; attainment asks whether some class member reaches that infimum.

Key takeaway: The best-in-class risk is the infimum of population risk over the restricted predictor class, while Bayes risk is the unrestricted infimum. Restriction can raise the risk, but exclusion of a Bayes predictor alone does not ensure a strictly positive gap or an attained best-in-class predictor.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

What is the difference between a best-in-class risk and a best-in-class predictor?
Why must $R_{\mathcal{G}}^*\geq R^*$ whenever $\mathcal{G}$ is a subset of the unrestricted predictor class?
In binary classification, what is the conditional error of predicting 0, and what is the conditional error of predicting 1?
In the alternating-label example, why can no threshold classifier achieve the Bayes risk?
Can a restricted class exclude every Bayes predictor and still have $R_{\mathcal{G}}^*=R^*$? Explain how the threshold example illustrates this.