Why Compare Two Conditional-Risk Infima?
A margin loss turns a real-valued score into a penalty that depends on the label and the score’s direction and magnitude. For classification, low surrogate risk is useful only if it favors scores on the Bayes-optimal side of zero. The key question is therefore not whether a particular score has low risk, but whether scores that point in the wrong direction can approach the smallest possible conditional risk.
Fix a feature value \(x\), and write \(\eta=P(Y=+1\mid X=x)\), where \(Y\in\{-1,+1\}\). The Bayes decision is positive when \(\eta>1/2\) and negative when \(\eta<1/2\). The earlier material on conditional zero-one risk establishes this decision rule. Here we compare the conditional risk of a margin loss over all real scores with its conditional risk over scores that do not point in the Bayes direction.
Conditional Risk and the Opposing-Score Set
If \(\eta>1/2\), then \(a_\eta>0\), so \(A_\eta=(-\infty,0]\): these scores are nonpositive even though the Bayes decision is positive. If \(\eta<1/2\), then \(a_\eta<0\), and \(A_\eta=[0,\infty)\): these scores are nonnegative even though the Bayes decision is negative. Zero is included because a zero score does not express either strict direction. The tied case \(\eta=1/2\) is omitted: there is no unique Bayes direction to oppose.
The infima are well-defined finite real numbers under the stated assumption on \(\phi\). The risk is nonnegative, and it has a finite value at every real score, so both infima are bounded below by zero and above by a finite value. Importantly, an infimum need not be attained by any score. The definition therefore uses infima, not presumed minimizing scores.
The unrestricted infimum is the best conditional surrogate risk available at that class probability. The constrained infimum is the best risk attainable while scores remain on the wrong side, including zero. Calibration says that the wrong side cannot get arbitrarily close to the unrestricted optimum. It does not require that the unrestricted infimum be attained, nor does it specify how large the gap is.
Worked Examples: Squared Margin Loss
Worked Example: Squared Margin Loss at a Fixed Class Probability
Take \(\phi(u)=(1-u)^2\). Then $$ C_\phi(\eta,s)=\eta(1-s)^2+(1-\eta)(1+s)^2. $$ Writing \(a=2\eta-1\), expand the two terms: $$ C_\phi(\eta,s)=s^2-2as+1=(s-a)^2+1-a^2. $$
The unrestricted infimum is attained at \(s=a\), with value \(1-a^2=4\eta(1-\eta)\). When \(\eta\ne 1/2\), the opposing set consists of scores satisfying \(as\le0\). The closest point in that set to \(a\) is zero, so the smallest value of \((s-a)^2\) over the set is \(a^2\), attained at \(s=0\). Thus $$ C_{\phi,\mathrm{opp}}^*(\eta)=1, \qquad C_{\phi,\mathrm{opp}}^*(\eta)-C_\phi^*(\eta)=a^2>0. $$ The loss is classification calibrated.
For a numerical check, let \(\eta=3/4\) and consider the opposing score \(s=-1/2\). Its positive-label squared loss is \((1-(-1/2))^2=(3/2)^2=9/4=2.25\); after weighting by the positive-label probability, that contribution is \((3/4)(9/4)=27/16\). The negative-label margin is \(-s=1/2\), giving squared loss \((1-1/2)^2=1/4\), and weighted contribution \((1/4)(1/4)=1/16\). Hence the full conditional risk is \(28/16=7/4=1.75\), not \(2.25\): \(2.25\) is the unweighted positive-label loss. The unrestricted infimum is \(1-(1/2)^2=3/4\), and the constrained infimum is \(1\), attained at zero. The gap is \(1-3/4=1/4\).
Two Consequences of the Definition
The constrained-infimum criterion has useful implications that do not depend on a formula for a particular loss. First, the two class probabilities on either side of one half have the same gap. Second, a positive gap gives a precise statement about sequences whose conditional risks approach the unrestricted infimum.
Proof. Direct substitution into the conditional risk gives $$ C_\phi(1-\eta,s)=(1-\eta)\phi(s)+\eta\phi(-s)=C_\phi(\eta,-s). $$ As \(s\) ranges over \(\mathbb{R}\), \(-s\) also ranges over \(\mathbb{R}\), so taking unrestricted infima yields \(C_\phi^*(1-\eta)=C_\phi^*(\eta)\). Also \(a_{1-\eta}=-a_\eta\), and $$ s\in A_{1-\eta} \ \Longleftrightarrow\ (-a_\eta)s\le0 \ \Longleftrightarrow\ a_\eta(-s)\le0 \ \Longleftrightarrow\ -s\in A_\eta. $$ The change of variable \(t=-s\) therefore maps the constrained set for \(1-\eta\) bijectively to the constrained set for \(\eta\). Using \(C_\phi(1-\eta,s)=C_\phi(\eta,-s)\) in the constrained infimum gives \(C_{\phi,\mathrm{opp}}^*(1-\eta)=C_{\phi,\mathrm{opp}}^*(\eta)\). Subtracting the equal unrestricted infima proves the result. \(\square\)
Proof. Let \(\delta=C_{\phi,\mathrm{opp}}^*(\eta)-C_\phi^*(\eta)>0\), and suppose \(C_\phi(\eta,s_m)\to C_\phi^*(\eta)\). By the definition of convergence, there is an index \(M\) such that for every \(m\ge M\), $$ C_\phi(\eta,s_m)<C_\phi^*(\eta)+\delta/2<C_{\phi,\mathrm{opp}}^*(\eta). $$ No \(s_m\) with \(m\ge M\) can belong to \(A_\eta\), since every score in \(A_\eta\) has risk at least \(C_{\phi,\mathrm{opp}}^*(\eta)\). This proves the first claim. For the converse, suppose the gap is zero. Then \(C_{\phi,\mathrm{opp}}^*(\eta)=C_\phi^*(\eta)\). By the defining property of an infimum, for every positive integer \(m\) there exists \(s_m\in A_\eta\) such that $$ C_\phi(\eta,s_m)<C_{\phi,\mathrm{opp}}^*(\eta)+1/m =C_\phi^*(\eta)+1/m. $$ Since \(C_\phi(\eta,s_m)\ge C_\phi^*(\eta)\), these risks converge to \(C_\phi^*(\eta)\), as required. \(\square\)
This sequence characterization explains why strict inequality matters. If the gap is zero, wrongly directed scores can approach the best risk, even if no such score achieves it. If the gap is positive, a sequence approaching the best risk must eventually leave the opposing set. This is a fixed-\(\eta\) statement; it does not claim that the gap is bounded away from zero uniformly as \(\eta\) approaches one half.
Worked Examples: Hinge and Misaligned Quadratic Losses
Worked Example: Hinge Loss at \(\eta=3/4\)
For the hinge loss \(\phi(u)=(1-u)_+=\max(0,1-u)\), consider \(\eta=3/4\), so \(a_\eta=1/2\) and the opposing scores satisfy \(s\le0\). The conditional risk is $$ C_\phi(3/4,s)=\frac34(1-s)_++\frac14(1+s)_+. $$ For \(s\le-1\), this equals \(\frac34(1-s)\), whose smallest value on that interval is \(3/2\), at \(s=-1\). For \(-1\le s\le0\), it equals \(1-\frac12s\), whose smallest value is \(1\), at \(s=0\). Thus the constrained infimum is \(1\).
For unrestricted scores, when \(-1\le s\le1\), the risk is \(1-\frac12s\), which decreases to \(1/2\) at \(s=1\). For \(s\ge1\), the risk is \(\frac14(1+s)\), which increases from \(1/2\). For \(s\le-1\), the risk is at least \(3/2\), as calculated above. Hence the unrestricted infimum is \(1/2\), and the conditional calibration gap is \(1-1/2=1/2>0\).
Worked Example: A Quadratic Loss That Favors the Wrong Sign
Consider instead \(\phi(u)=(1+u)^2\). At \(\eta=3/4\), its conditional risk is $$ C_\phi(3/4,s)=\frac34(1+s)^2+\frac14(1-s)^2 =(s+1/2)^2+\frac34. $$ The unrestricted infimum is \(3/4\), attained at \(s=-1/2\). But \(\eta=3/4\) has Bayes-positive direction, so the opposing set is \(s\le0\), which includes \(-1/2\). The constrained and unrestricted infima are both \(3/4\), and the gap is zero. This loss is not classification calibrated.
The example shows why the infimum comparison is substantive: merely having a loss that is finite and nonnegative does not ensure that minimizing its conditional risk favors the Bayes sign. Here the conditional optimum itself has the wrong sign.
Common Pitfall and Scope
Common mistake: replacing the constrained infimum with the risk at one selected wrong-sign score, then concluding that calibration holds if that score has larger risk than the unrestricted infimum. That checks only one point. Other scores in the opposing set might have much smaller risk, or a sequence of them might approach the unrestricted infimum. The required comparison is between the infimum over the entire opposing set and the unrestricted infimum.
A second error is to use “minimum” automatically. An infimum is the greatest lower bound and need not be achieved. For example, a loss may have risks decreasing toward a limiting value as \(s\) tends to infinity without ever attaining that value. The sequence result above is why the definition remains meaningful in that setting.
Finally, the calibration gap is conditional on a fixed class probability. It says whether a wrong-direction score can be conditionally optimal, and whether it can approach the conditional optimum arbitrarily closely. It does not by itself give a population regret bound, a rate of convergence, or a guarantee about probability estimation. Those questions require additional results beyond this definition.
Check Your Understanding
Use the definitions and calculations above to answer the following questions.
- For \(\eta=0.2\), write the opposing-score set \(A_\eta\) explicitly and explain why its scores oppose the Bayes direction.
- Why does the definition use an infimum rather than assuming a conditional-risk minimizer exists?
- For the squared margin loss \(\phi(u)=(1-u)^2\), what are the unrestricted infimum, constrained infimum, and gap when \(\eta=1/4\)?
- What does a zero calibration gap imply about the existence of a sequence of opposing scores?
- In the squared-loss calculation at \(\eta=3/4\), distinguish the positive-label loss at \(s=-1/2\) from its probability-weighted contribution and from the full conditional risk.