Conditional Risk at the Decision Boundary
For binary classification, the posterior probability \(\eta=P(Y=1\mid X=x)\) determines the best label at a fixed feature value \(x\). The boundary is \(\eta=1/2\): on either side, the optimal label changes. Near this boundary, a small error in the score's sign can cause a small classification penalty, while the surrogate loss also has a characteristic excess-risk cost. Calculating both quantities directly makes that distinction precise.
We use labels \(Y\in\{-1,+1\}\) and a real-valued score \(t\). The prediction is \(+1\) when \(t\geq 0\), and \(-1\) when \(t<0\); in particular, a score of zero predicts \(+1\). Write \(\eta=P(Y=+1\mid X=x)\). All calculations below are conditional on this fixed \(x\), so \(\eta\) is a number in \([0,1]\).
The Bayes label is \(+1\) for \(\eta>1/2\), \(-1\) for \(\eta<1/2\), and either label for \(\eta=1/2\). Consequently, a score with the wrong sign has classification regret \(|2\eta-1|\). A score on the correct side, including zero when the convention predicts the Bayes label, has classification regret zero. At the boundary every prediction has zero classification regret.
Three Conditional Risks and Their Minimizers
We use the hinge loss \(\phi_{\mathrm h}(u)=(1-u)_+\), the logistic loss \(\phi_{\mathrm l}(u)=\log(1+e^{-2u})\), and the exponential loss \(\phi_{\mathrm e}(u)=e^{-u}\). The factor of \(2\) in the logistic margin is part of this score convention; it gives the minimizer \(t=\tfrac12\log(\eta/(1-\eta))\). For \(\eta=0\) or \(1\), formulas involving a logarithm are interpreted by their limiting behavior.
Worked Example: Hinge Risk on the Three Score Regions
For \(-1\leq t\leq1\), both hinge terms are linear, and $$ C_{\mathrm h}(t;\eta)=\eta(1-t)+(1-\eta)(1+t)=1+(1-2\eta)t. $$ For \(t\geq1\), the positive-label loss is zero and \(C_{\mathrm h}(t;\eta)=(1-\eta)(1+t)\), which is nondecreasing. For \(t\leq-1\), \(C_{\mathrm h}(t;\eta)=\eta(1-t)\), which decreases as \(t\) increases. Thus for \(\eta>1/2\) a minimum occurs at \(t=1\), with value \(2(1-\eta)\); for \(\eta<1/2\) it occurs at \(t=-1\), with value \(2\eta\). At \(\eta=1/2\), every \(t\in[-1,1]\) minimizes the risk, whose value is \(1\).
Worked Example: Logistic Conditional Risk
The conditional logistic risk is $$ C_{\mathrm l}(t;\eta)=\eta\log(1+e^{-2t})+(1-\eta)\log(1+e^{2t}). $$ Differentiation gives $$ C_{\mathrm l}'(t;\eta)=2\left(\frac{e^{2t}}{1+e^{2t}}-\eta\right),\qquad C_{\mathrm l}''(t;\eta)=\frac{4e^{2t}}{(1+e^{2t})^2}>0. $$ For \(0<\eta<1\), strict convexity and the zero derivative give the unique minimizer \(t^*=\tfrac12\log(\eta/(1-\eta))\). At this score, \(e^{2t^*}=\eta/(1-\eta)\), so \(1+e^{2t^*}=1/(1-\eta)\) and \(1+e^{-2t^*}=1/\eta\). Substitution yields $$ \inf_t C_{\mathrm l}(t;\eta)=-\eta\log\eta-(1-\eta)\log(1-\eta). $$ At \(\eta=1/2\), this minimizer is zero and the minimum is \(\log 2\).
Worked Example: Exponential Conditional Risk
Here \(C_{\mathrm e}(t;\eta)=\eta e^{-t}+(1-\eta)e^t\). For \(0<\eta<1\), $$ C_{\mathrm e}'(t;\eta)=-\eta e^{-t}+(1-\eta)e^t,\qquad C_{\mathrm e}''(t;\eta)=\eta e^{-t}+(1-\eta)e^t>0. $$ Setting the first derivative to zero gives \(e^{2t^*}=\eta/(1-\eta)\), hence \(t^*=\tfrac12\log(\eta/(1-\eta))\). At this score, the two terms in the risk are each \(\sqrt{\eta(1-\eta)}\), so the minimum is \(2\sqrt{\eta(1-\eta)}\). At \(\eta=1/2\), \(t^*=0\) and the minimum is \(1\).
Wrong-Side Excess Risk
A wrong-side score is one whose predicted class disagrees with the Bayes label. When \(\eta>1/2\), such scores have \(t<0\); when \(\eta<1/2\), they have \(t\geq0\), because zero predicts \(+1\). The infimum over a wrong side need not be attained: the wrong side can be open at zero. The following result gives the infimum, which is the relevant conditional surrogate cost of insisting on the wrong decision.
To prove this, first consider \(\eta>1/2\). For hinge loss, the risk on \([-1,0]\) is \(1+(1-2\eta)t\), which decreases as \(t\) increases; for \(t\leq-1\), the risk \(\eta(1-t)\) is at least its value \(2\eta\) at \(-1\), and \(2\eta>1\). Thus the wrong-side infimum is the limit \(1\) as \(t\uparrow0\). Subtracting the unrestricted minimum \(2(1-\eta)\) gives \(2\eta-1\). For \(\eta<1/2\), the wrong-side region includes zero; hinge risk is minimized there at zero, with value \(1\). Subtracting \(2\eta\) gives \(1-2\eta\). Both cases yield \(|2\eta-1|\).
For logistic and exponential losses, each conditional risk is minimized at \(t^*=\tfrac12\log(\eta/(1-\eta))\), which has the Bayes sign. Strict convexity implies that, on the wrong side, the infimum is the boundary value at zero (or its limit from the open side). Those values are \(C_{\mathrm l}(0;\eta)=\log2\) and \(C_{\mathrm e}(0;\eta)=1\). Subtracting the respective unrestricted minima derived above gives the stated expressions. When \(\eta<1/2\), zero is included among wrong predictions and attains the boundary value; when \(\eta>1/2\), it is only approached from below. This distinction changes attainment, not the infimum. \(\square\)
Numerical Comparison Just Below, At, and Above One Half
Take \(\eta=0.49\), \(0.50\), and \(0.51\). The posterior odds at \(0.49\) give $$ \frac12\log\left(\frac{0.49}{0.51}\right)\approx -0.0200027, $$ and at \(0.51\) the minimizing score is \(+0.0200027\). For exponential loss at either off-boundary value, the minimum is $$ 2\sqrt{0.49(0.51)}=2\sqrt{0.2499}=\sqrt{0.9996}\approx0.99979998. $$ The logistic minimum is the binary entropy \(-0.49\log(0.49)-0.51\log(0.51)\approx0.69294717\). Thus the wrong-side excess costs are approximately \(0.02\) for hinge, \(0.00020001\) for logistic, and \(0.00020002\) for exponential.
| \(\eta\) | Bayes label | Classification regret at \(t=0\) | Surrogate regret at \(t=0\): hinge / logistic / exponential |
|---|---|---|---|
| 0.49 | \(-1\) | 0.51 − 0.49 = 0.02 | 0.02 / 0.00020001 / 0.00020002 |
| 0.50 | Either label | 0 | 0 / 0 / 0 |
| 0.51 | \(+1\) | 0 | 0.02 / 0.00020001 / 0.00020002 |
At \(\eta=0.49\), a zero score predicts \(+1\), so it is wrong: its classification risk is \(1-0.49=0.51\), while the Bayes risk is \(0.49\), giving regret \(0.51-0.49=0.02\). At \(\eta=0.51\), zero predicts the Bayes label, so classification regret is zero. Nevertheless, its logistic and exponential surrogate regrets are positive: the risk at zero is not minimal, since their minimizing score is positive. Hinge risk at zero also exceeds its minimum by \(0.02\), although the prediction is correct. A correct sign alone does not imply zero surrogate regret.
At exactly \(\eta=1/2\), every score has zero classification regret because the two labels have equal conditional risk. The logistic and exponential risks attain their minima at zero. Hinge risk is constant at \(1\) throughout \([-1,1]\), so zero is also a minimizer. For hinge loss, this flat interval illustrates that a conditional minimizer need not be unique.
How the Costs Vanish Near the Boundary
Write \(\eta=1/2+\delta\). The classification regret for a wrong prediction is \(2|\delta|\). For hinge loss, the wrong-side surrogate excess is exactly the same quantity. For logistic loss, the wrong-side excess is $$ \left(\frac12+\delta\right)\log(1+2\delta)+\left(\frac12-\delta\right)\log(1-2\delta), $$ and for exponential loss it is \(1-\sqrt{1-4\delta^2}\). For small \(\delta\), Taylor expansion at zero gives $$ \text{logistic excess}=2\delta^2+O(\delta^4),\qquad \text{exponential excess}=2\delta^2+O(\delta^4). $$ The expansions apply for \(|\delta|<1/2\). For example, the exponential expression follows by expanding \(\sqrt{1-z}=1-z/2+O(z^2)\) at \(z=4\delta^2\). The logistic expression is even in \(\delta\); its quadratic coefficient is \(2\), and the next nonzero term has order \(\delta^4\).
Thus the hinge wrong-side excess vanishes linearly in the distance from the boundary, while the logistic and exponential wrong-side excesses vanish quadratically. This difference is useful when interpreting conditional calibration calculations: very near the boundary, the surrogate cost of a wrong sign can be much smaller than the classification regret. The exact formulas, rather than a numerical comparison at one posterior value, determine that behavior.
Common Pitfall
A common error is to treat a score of zero as automatically having zero classification regret, or to subtract the Bayes risk incorrectly. The sign-at-zero convention matters: here zero predicts \(+1\). At \(\eta=0.49\), that is the wrong label, and the regret is not \(1-0.49\) by itself; \(1-0.49=0.51\) is the risk of predicting \(+1\), so the regret is \(0.51-0.49=0.02\). Conversely, at \(\eta=0.51\), zero predicts the correct label and has zero classification regret, even though its surrogate excess risk is positive. Always identify the prediction first, then subtract the Bayes risk, and keep classification regret distinct from surrogate regret.
Check Your Understanding
Use the stated sign-at-zero convention and distinguish classification risk from surrogate risk.
- For \(\eta=0.48\), what label does a score of zero predict, and what is its classification regret?
- At \(\eta=0.51\), why can a score of zero have zero classification regret but positive logistic surrogate regret?
- Compute the exponential conditional minimum at \(\eta=0.25\).
- For \(\eta>1/2\), explain why the hinge wrong-side infimum is approached at zero rather than attained by a negative score.
- As \(\delta\) tends to zero, compare the order of the hinge and logistic wrong-side excess risks when \(\eta=1/2+\delta\).