Why Compare Conditional Risks?
A classifier makes a prediction using the observed feature value. To decide which label to predict at a particular value \(x\), we can compare the probabilities of the two possible mistakes given \(X=x\). This turns the classification decision into a simple risk comparison: predicting zero is costly when the true label is one, while predicting one is costly when the true label is zero.
The binary regression function records the conditional probability of class one. The central question here is how that probability determines the conditional zero-one risk of each possible prediction, and why choosing the prediction with the smaller risk produces the Bayes decision.
When \(X\) has a continuous distribution, a single point \(x\) may have probability zero. In that setting, \(P(Y=1\mid X=x)\) is understood through a regular conditional distribution, not as a ratio of probabilities involving the event \(\{X=x\}\). A measurable version of \(\eta\) is defined \(P_X\)-almost everywhere; its values on a \(P_X\)-null set can be assigned arbitrarily without changing overall risk.
Writing the Risk in Terms of \(\eta(x)\)
There are only two possible predictions. If the classifier predicts zero, it makes an error exactly when \(Y=1\). If it predicts one, it makes an error exactly when \(Y=0\). Therefore the two conditional risks are
This pair of expressions contains the entire pointwise decision problem. The classifier should predict the label whose conditional probability is larger: predicting zero has risk \(\eta(x)\), and predicting one has risk \(1-\eta(x)\). The smaller of the two risks is the conditional risk achieved by the best prediction at \(x\).
The threshold is \(1/2\) because zero-one loss assigns the same penalty to either kind of mistake. At equality, the conditional probabilities of the two labels are the same, so choosing either label has the same conditional risk. A convention such as predicting one at equality makes the rule definite, but does not improve or worsen its risk.
Worked Conditional-Risk Comparisons
Worked Example: A Feature Value Favoring Class One
At a particular feature value \(x\), suppose \(\eta(x)=0.73\). Calculate the conditional risks of both predictions and identify a minimizer.
If the prediction is zero, the error occurs when \(Y=1\), so
If the prediction is one, the error occurs when \(Y=0\), so
Since \(0.27<0.73\), predicting one has the lower conditional risk. The fact that \(\eta(x)=0.73\) is a probability of class one does not mean the risk of predicting one is \(0.73\); that prediction errs on class zero, whose conditional probability is \(0.27\).
Worked Example: The Threshold Tie
Suppose \(\eta(x)=0.5\). For prediction zero,
For prediction one,
Both deterministic predictions have the same conditional risk. A tie is not a situation in which the classifier has failed to choose the uniquely correct answer: there is no uniquely better label under zero-one loss at this \(x\).
Worked Example: A Continuous Feature and Overall Risk
Let \(X\) be uniform on \([0,1]\), and suppose \(Y\mid X=x\) is Bernoulli with success probability \(\eta(x)=x\). Thus \(P(Y=1\mid X=x)=x\). For \(x<1/2\), predicting zero has risk \(x\), while predicting one has risk \(1-x\); the smaller risk is \(x\). For \(x>1/2\), the smaller risk is \(1-x\), achieved by predicting one. At \(x=1/2\), both risks are \(1/2\).
The threshold rule is therefore \(g^*(x)=0\) for \(x<1/2\) and \(g^*(x)=1\) for \(x>1/2\), with either choice at \(1/2\). Its overall risk is the average of its conditional risks over the uniform distribution of \(X\):
The integral uses the uniform density, which is one on \([0,1]\). The single tie point \(x=1/2\) has probability zero here, so the choice made there does not affect the overall risk.
Why Pointwise Minimization Gives the Bayes Rule
The pointwise comparison is not merely a convenient heuristic. It proves optimality among measurable classifiers. A classifier is a measurable function \(g\) that maps feature values to labels in \(\{0,1\}\). Its overall zero-one risk is \(R(g)=P(g(X)\ne Y)\). For any such \(g\), conditioning on \(X\) gives
The equality follows from the tower property of conditional expectation applied to the error indicator. The conditional risk at \(X\) equals \(\eta(X)\) when \(g(X)=0\), and \(1-\eta(X)\) when \(g(X)=1\).
Proof. Because \(\eta\) is measurable, the set \(\{x:\eta(x)\ge 1/2\}\) is measurable, so \(g^*\) is a measurable classifier. Fix any measurable classifier \(g\). At every feature value where \(\eta(x)\) is defined, \(g^*(x)\) chooses an action with the smaller of \(R_x(0)\) and \(R_x(1)\); at a tie, both actions have the same risk. Hence
This inequality holds \(P_X\)-almost everywhere. Both sides are bounded between zero and one, so they are integrable. Taking expectations and using the conditional-risk representation yields
Since this holds for every measurable \(g\), \(g^*\) minimizes overall zero-one risk. Changing the rule only where \(\eta(x)=1/2\) preserves the pointwise conditional risk there and therefore preserves the overall risk. \(\square\)
Randomized Predictions and the Role of Ties
A randomized classifier can choose prediction one with probability \(q(x)\in[0,1]\), conditional on the observed \(x\). Its conditional risk is a weighted average of the risks of the two deterministic actions:
This expression is linear in \(q(x)\). If \(\eta(x)<1/2\), then \(R_x(0)<R_x(1)\), and every mixture that puts positive probability on prediction one has risk at least as large as always predicting zero. If \(\eta(x)>1/2\), always predicting one is at least as good as any mixture. If \(\eta(x)=1/2\), then
for every \(q(x)\in[0,1]\). Thus randomization cannot improve conditional risk. It can change the selected prediction, but away from a tie it cannot lower the minimum; at a tie, every randomized choice has the same risk as either deterministic choice.
Common Mistakes
- Using \(\eta(x)\) as the risk of predicting one. \(\eta(x)\) is the conditional probability that the true label is one. Predicting one makes an error when the true label is zero, so its conditional risk is \(1-\eta(x)\).
- Treating equality as a strict preference. At \(\eta(x)=1/2\), the two conditional risks are equal. A threshold convention can select a label, but neither label is strictly better under zero-one loss.
- Reading a pointwise rule as a guarantee at every possible \(x\). A conditional probability at a zero-probability feature value depends on the chosen version of the conditional distribution. The risk-minimization claim concerns values of \(x\) up to \(P_X\)-null sets.
- Assuming a more complicated decision helps by itself. Once the conditional risks of both labels have been compared, randomization cannot produce a lower value than the smaller of those two risks.
Check Your Understanding
For each question, use the conditional risks of the two possible labels.
- If \(\eta(x)=0.18\), calculate \(R_x(0)\) and \(R_x(1)\). Which prediction minimizes conditional risk?
- At \(\eta(x)=1/2\), do the deterministic predictions have equal or different conditional risks? Explain.
- For a randomized classifier with probability \(q(x)\) of predicting one, write its conditional risk and explain why it cannot beat both deterministic actions.
- In the continuous-feature example with \(X\) uniform on \([0,1]\) and \(\eta(x)=x\), what prediction minimizes risk at \(x=0.3\), and what is its conditional risk?
- Why is the conditional-risk minimization claim stated up to \(P_X\)-null sets?