Common Errors in Calibration Arguments

Why Calibration Arguments Need Careful Auditing

Calibration arguments connect a real-valued score and its surrogate loss to the decisions made by a classifier. Several different quantities appear in that connection: a derivative at the decision boundary, a constrained conditional infimum, surrogate excess risk, classification excess risk, and an inverse function. Treating any two of these as interchangeable can produce an incorrect guarantee.

We will use binary labels $Y\in\{-1,+1\}$, a score $s\in\mathbb R$, and a margin loss $\phi:\mathbb R\to\mathbb R$. For a fixed feature value $x$, write $\eta=P(Y=+1\mid X=x)$. The conditional surrogate risk is the expected loss at that $x$; the score predicts $+1$ when $s\geq 0$ and $-1$ when $s<0$. This tie convention will matter when describing classification errors.

Definition: The conditional surrogate risk and its unrestricted infimum are $$ C_\eta(s)=\eta\phi(s)+(1-\eta)\phi(-s), \qquad C_\eta^*=\inf_{s\in\mathbb R} C_\eta(s). $$ For $\eta>1/2$, define the closed wrong-side conditional gap as $\inf_{s\leq 0}C_\eta(s)-C_\eta^*$; for $\eta<1/2$, define it as $\inf_{s\geq 0}C_\eta(s)-C_\eta^*$. The closed constraints include the boundary score $s=0$, regardless of the classifier’s tie convention.

The Derivative Criterion Has Hypotheses

A familiar criterion says that a convex margin loss is classification-calibrated when its derivative at zero is negative. It is not a rule to apply to any loss-like function without checking assumptions. In the form used here, $\phi$ must be finite and convex on $\mathbb R$, and differentiable at zero. Calibration means that for every $\eta\neq 1/2$, the wrong-side conditional gap is strictly positive.

Theorem: If $\phi$ is finite, convex, and differentiable at zero, then it is classification-calibrated in the stated sense if and only if $\phi'(0)<0$.

Proof. Differentiate the conditional risk at zero:

C_\eta'(0)=\eta\phi'(0)-(1-\eta)\phi'(0) =(2\eta-1)\phi'(0).

Suppose first that $\phi'(0)<0$ and $\eta>1/2$. Then $C_\eta'(0)<0$. Since $C_\eta$ is convex, its derivative, where defined, is nondecreasing; consequently it is negative to the left of zero. Thus $C_\eta$ decreases as $s$ moves right toward zero on $s\leq0$, and $\inf_{s\leq0}C_\eta(s)=C_\eta(0)$. The negative derivative at zero also means there is some $s_0>0$ with $C_\eta(s_0)<C_\eta(0)$. Therefore $C_\eta^*<\inf_{s\leq0}C_\eta(s)$, so the gap is positive. For $\eta<1/2$, the same argument with signs reversed gives a positive gap on $s\geq0$.

Conversely, if $\phi'(0)=0$, then $C_\eta'(0)=0$ for every $\eta$. Convexity makes zero a global minimizer of each $C_\eta$, so the wrong-side gap is zero. If $\phi'(0)>0$, take $\eta>1/2$. Then $C_\eta'(0)>0$: moving a little to the left lowers the risk, while convexity implies $C_\eta(s)\geq C_\eta(0)$ for $s\geq0$. The unrestricted infimum is therefore attained as an infimum on the wrong side $s\leq0$, and the gap is zero. Hence calibration requires $\phi'(0)<0$. $\square$

The proof also shows why a derivative test cannot be detached from its assumptions. For a nondifferentiable loss, a single value called “the derivative” may not exist; the relevant one-sided slopes or subgradients must be analyzed instead. Nor does the criterion itself supply a numerical rate: it establishes strict separation, not a particular calibration function.

Worked Example: A Zero Derivative Does Not Calibrate

Take the finite convex margin loss $\phi(u)=u^2$. Its derivative at zero is zero. Direct calculation gives

C_\eta(s)=\eta s^2+(1-\eta)(-s)^2=s^2.

Thus $C_\eta^*=0$, attained at $s=0$, for every $\eta$. When $\eta=3/4$, for example, the constrained infimum over $s\leq0$ is also zero. The wrong-side gap is zero, not positive. The loss is convex, but the derivative criterion correctly rejects it as calibrated.

Do Not Drop the Boundary Score

A frequent calculation error is to use an interior formula for a piecewise loss outside the interval where it applies. Another is to replace a closed wrong-side constraint by a strict one without checking what happens at zero. The boundary can determine the constrained infimum, even when the classifier assigns zero to one class by convention.

Worked Example: The Hinge Loss at a Boundary

Let $\phi(u)=(1-u)_+$, where $a_+=\max(a,0)$, and set $\eta=3/4$. Evaluating each hinge term in its correct range gives

C_{3/4}(s)= \begin{cases} \frac34(1-s), & s\leq -1,\\[2pt] 1-\frac12s, & -1\leq s\leq 1,\\[2pt] \frac14(1+s), & s\geq 1. \end{cases}

On $s\leq0$, the interval $[-1,0]$ has risk $1-\tfrac12s$, whose minimum is $1$ at zero. On $s\leq-1$, the risk is $\tfrac34(1-s)$, whose minimum is $3/2$ at $-1$. Therefore $\inf_{s\leq0}C_{3/4}(s)=1$. On the unrestricted domain, the middle piece decreases up to $s=1$, and the right piece increases from $s=1$; the global minimum is $C_{3/4}(1)=1/2$. The wrong-side gap is $1-1/2=1/2$.

The expression $1-\tfrac12s$ is valid only on $[-1,1]$, not on all $s\leq0$. For instance, at $s=-2$ the correct risk is $\tfrac34(3)=9/4$, not $1-\tfrac12(-2)=2$. The constrained infimum happens still to be $1$, but that conclusion must come from the piecewise calculation.

At $s=0$, the chosen classifier predicts $+1$, which is correct for $\eta=3/4$. The closed constraint $s\leq0$ nevertheless includes zero. This makes the gap a conservative boundary-based quantity; it does not claim that every score in the constrained set is a classification error under every tie convention.

Surrogate Excess and Classification Excess Are Different

For a classifier $g$, classification excess risk measures its additional probability of error relative to the Bayes classifier. At a feature value with $\eta\neq1/2$, predicting the less probable label incurs conditional classification excess $|2\eta-1|$; predicting the more probable label incurs zero. In contrast, conditional surrogate excess at score $s$ is $C_\eta(s)-C_\eta^*$. It depends on the loss and on the score’s distance from its conditional minimizer, not just on which label the score predicts.

Worked Example: Two Excess Risks for the Same Score

Continue with the hinge loss, $\eta=3/4$, and score $s=-1$. Since the score predicts $-1$, the conditional classification excess is

|2\eta-1|=\left|2\cdot\frac34-1\right|=\frac12.

The piecewise calculation gives $C_{3/4}(-1)=3/2$, while $C_{3/4}^*=1/2$. Hence the conditional surrogate excess is $3/2-1/2=1$, not $1/2$. These quantities can be compared through a calibration inequality, but they are not equal and should not be substituted for one another.

From Conditional Gaps to a Valid Population Bound

The calibration function records how much surrogate excess is forced by a wrong-side score. For a symmetric binary margin loss, define, for $0\leq\theta\leq1$,

\delta_\phi(\theta) = \inf_{s\leq0}C_{(1+\theta)/2}(s) - C_{(1+\theta)/2}^*.

Here $\theta=|2\eta-1|$ is the magnitude of the conditional class imbalance. The definition uses the closed half-line; for $\eta<1/2$, reflection $s\mapsto-s$ gives the corresponding bound on $s\geq0$. At $\theta=0$, the two classes are tied and the gap is zero.

Theorem: Let $f(X)$ be a measurable score with finite surrogate excess risk, and let $R(f)$ be the classification risk of its induced classifier. Suppose a convex, nondecreasing function $\delta:[0,1]\to[0,\infty)$, with $\delta(0)=0$, is a lower bound on the wrong-side conditional gap at every $\theta$. Then $$ \delta\bigl(R(f)-R^*\bigr) \leq R_\phi(f)-R_\phi^*. $$ If $\delta$ is nondecreasing, this implies $R(f)-R^*\leq\delta^{\leftarrow}(R_\phi(f)-R_\phi^*)$, where $\delta^{\leftarrow}(u)=\sup\{r\in[0,1]:\delta(r)\leq u\}$.

Proof. Write $m(X)=|2\eta(X)-1|$ times the indicator that $f(X)$ predicts the less probable class. This is the conditional classification excess, so $E[m(X)]=R(f)-R^*$. If the score predicts the wrong class, it lies in the corresponding closed wrong-side half-line, and the definition of the gap gives

C_{\eta(X)}(f(X))-C_{\eta(X)}^* \geq \delta(m(X)).

If the score predicts the Bayes class, then $m(X)=0$, and the same inequality follows because conditional surrogate excess is nonnegative and $\delta(0)=0$. Taking expectations and applying Jensen’s inequality, which is valid because $\delta$ is convex and $m(X)\in[0,1]$, yields

R_\phi(f)-R_\phi^* \geq E[\delta(m(X))] \geq \delta(E[m(X)]) =\delta(R(f)-R^*).

Since $\delta$ is nondecreasing, the displayed inequality places $R(f)-R^*$ in the set $\{r:\delta(r)\leq R_\phi(f)-R_\phi^*\}$, giving the generalized-inverse bound. $\square$

This proof identifies two assumptions that are easy to omit. First, the pointwise conditional inequality must apply to the score’s actual decision, including the boundary convention. Second, Jensen’s inequality requires a convex function. If the pointwise gap is not convex, one cannot simply move $\delta$ outside the expectation; a suitable convex lower bound or a different argument is needed.

Worked Example: Inverting the Exponential Calibration Bound

For the exponential loss $\phi(u)=e^{-u}$,

C_\eta(s)=\eta e^{-s}+(1-\eta)e^s.

Differentiating and setting the derivative to zero gives $e^{2s}=\eta/(1-\eta)$, so $s^*=\tfrac12\log(\eta/(1-\eta))$. Substitution yields $C_\eta^*=2\sqrt{\eta(1-\eta)}$. For $\eta>1/2$, the derivative at zero is $1-2\eta<0$; convexity therefore makes the constrained infimum over $s\leq0$ equal to $C_\eta(0)=1$. With $\theta=2\eta-1$, the calibration function is

\delta_\phi(\theta) =1-2\sqrt{\eta(1-\eta)} =1-\sqrt{1-\theta^2}.

This function is nondecreasing and convex on $[0,1]$. Solving $u=1-\sqrt{1-\theta^2}$ for $\theta$ gives the correct inverse:

\theta=\sqrt{2u-u^2}.

For a single feature value with $\eta=3/4$ and score $s=-0.1$, the classification excess is $\theta=1/2$. The conditional surrogate risk is

C_{3/4}(-0.1) =\frac34e^{0.1}+\frac14e^{-0.1} \approx 1.05509, \qquad C_{3/4}^*=\frac{\sqrt3}{2}\approx0.86603.

The surrogate excess is approximately $0.18906$. The inverse bound gives classification excess at most $\sqrt{2(0.18906)-(0.18906)^2}\approx0.585$, which is valid but loose relative to the actual value $0.5$. Replacing the inverse by $\delta(u)$, or claiming the classification excess is at most $0.18906$, would be an unjustified inversion.

Common Errors to Check

Using the derivative criterion without its conditions. Check convexity, finiteness, and differentiability at zero before applying the stated criterion. For a nonsmooth loss, analyze the appropriate one-sided slopes or subgradients.
Dropping $s=0$ or extending a piecewise formula. State the constrained set explicitly, include the boundary when using a closed wrong-side gap, and verify every piecewise expression on its stated interval.
Calling surrogate excess “classification regret.” Compute each quantity from its definition. One is a loss difference in score space; the other is an excess probability of classification error.
Inverting without checking direction or shape. From $\delta(R)\leq\varepsilon$, use the appropriate nondecreasing generalized inverse. A population inequality also needs a justified passage from conditional to expected risk, such as Jensen’s inequality for a convex $\delta$.

Key takeaway: A sound calibration argument keeps the conditional constraint, the score-zero convention, the two excess risks, and the inverse function explicit. Each step must be justified under its own assumptions.

Check Your Understanding

For each question, identify the condition or calculation needed before drawing a conclusion.

For a finite convex margin loss differentiable at zero with $\phi'(0)=0$, what does the derivative criterion imply about the conditional minimizer at zero?
In the hinge-loss example with $\eta=3/4$, why must $s=-2$ be evaluated using a different piece of the risk formula than $s=0$?
With $\eta=3/4$ and hinge score $s=-1$, calculate the classification excess and the surrogate excess separately.
What role does convexity of $\delta$ play in deriving a population bound from conditional calibration gaps?
If $\delta(R)\leq\varepsilon$ and $\delta$ is nondecreasing, which inverse operation gives an upper bound on $R$?