Calibration Functions Near Zero

Why the Behavior Near Zero Matters

Calibration functions quantify how much conditional surrogate excess risk is forced by a classification error. Their behavior near zero is especially important: small classification regret corresponds to conditional class probabilities near the decision boundary, and the calibration function describes the surrogate cost in that regime. A function that grows linearly gives a different rate conversion from one that grows quadratically.

We compare three margin losses: hinge, logistic, and exponential. Write \(\eta=P(Y=1\mid X=x)\), \(\theta=2\eta-1\), and let \(u\in\mathbb R\) be a margin score. Thus \(\theta\in[-1,1]\), and the Bayes-optimal sign is the sign of \(\theta\). The analysis below uses \(t=|\theta|\), the magnitude of the conditional class imbalance.

The definition uses infima rather than requiring minimizers. That distinction matters at \(\eta=0\) or \(\eta=1\): for logistic and exponential losses, the unrestricted infimum is approached as the score tends to an infinite value and is not attained at any finite score.

The Three Calibration Functions

For \(\theta\geq0\), the constrained scores satisfy \(u\leq0\); symmetry gives the same calibration function when \(\theta<0\). The three losses and their calibration functions are as follows.

Here is the minimization behind the formulas. For the hinge loss, when \(\theta\geq0\), the unrestricted minimum is \(1-\theta\), attained at \(u=1\) (also at any \(u\geq1\) if \(\eta=1\)). On \(u\leq0\), the conditional risk is minimized at \(u=0\) and equals \(1\). Their difference is \(\theta\).

For logistic loss and \(0<\eta<1\), differentiating the conditional risk gives the finite minimizer \(u=\log(\eta/(1-\eta))\). Substitution gives the unrestricted minimum \(-\eta\log\eta-(1-\eta)\log(1-\eta)\). On \(u\leq0\), the risk is minimized at \(u=0\), with value \(\log2\), when \(\theta\geq0\). If \(\eta=1\), the unrestricted risk is \(\log(1+e^{-u})\): its infimum is \(0\) as \(u\to+\infty\), not an attained minimum. If \(\eta=0\), the symmetric statement holds as \(u\to-\infty\). Taking these infima yields the stated formula, including its endpoint values.

For exponential loss, the conditional risk is \(\eta e^{-u}+(1-\eta)e^u\). For \(0<\eta<1\), its unrestricted minimum occurs at \(u=\frac12\log(\eta/(1-\eta))\), with value \(2\sqrt{\eta(1-\eta)}=\sqrt{1-\theta^2}\). On \(u\leq0\) it is minimized at zero, with value \(1\), when \(\theta\geq0\). At \(\eta=1\), the risk is \(e^{-u}\), whose infimum is \(0\) as \(u\to+\infty\), but which has no finite minimizer; \(\eta=0\) is symmetric. Thus the endpoint calibration value is also given by the formula above.

Expansions Near Zero

The hinge calibration function has no curvature to expand: it is exactly linear. Logistic and exponential calibration functions are smooth around zero and begin with a quadratic term. The following derivations identify both the leading powers and the next corrections.

For logistic loss, differentiating its exact calibration formula for \(|t|<1\) gives \(\delta_{\mathrm L}'(t)=\frac12\log((1+t)/(1-t))\). The right side is \(\operatorname{artanh}(t)\), whose geometric-series expansion is \(t+t^3/3+t^5/5+\cdots\). Integrating from zero to \(t\), using \(\delta_{\mathrm L}(0)=0\), gives \(\delta_{\mathrm L}(t)=t^2/2+t^4/12+t^6/30+\cdots\). The power series converges for \(|t|<1\); in particular, truncating after the fourth-order term leaves a remainder \(O(t^6)\) near zero.

For exponential loss, apply the Taylor expansion of \(\sqrt{1-z}\) at \(z=0\): \(\sqrt{1-z}=1-z/2-z^2/8+O(z^3)\). Putting \(z=t^2\) gives \(1-\sqrt{1-t^2}=t^2/2+t^4/8+O(t^6)\). The hinge identity \(\delta_{\mathrm H}(t)=t\) needs no approximation. This proves the stated expansions. \(\square\)

From Local Powers to Regret-Rate Exponents

The conditional classification regret at a feature value is \(t\) when the score has the wrong Bayes sign, and zero otherwise. By the constrained-infimum definition, a wrong-sign score has conditional surrogate excess at least \(\delta_\phi(t)\). A correct-sign score has zero classification regret. Thus the calibration function converts a pointwise classification regret into a lower bound on surrogate excess.

This gives the general exponent rule. If \(\delta_\phi(t)\) behaves like \(c t^q\) near zero, with \(c>0\), then a small conditional surrogate excess \(\varepsilon\) permits classification regret of order \(\varepsilon^{1/q}\), not \(\varepsilon\), in the small-regret regime. For hinge loss \(q=1\); for logistic and exponential losses \(q=2\). This local rule can be made into a population bound here because the relevant calibration functions are convex on \([0,1]\).

To justify the population statement, let \(R(x)\) denote the conditional classification regret. It is either zero or \(t(x)=|2\eta(x)-1|\). At every \(x\), the conditional surrogate excess is at least \(\delta_\phi(R(x))\): this follows from the constrained infimum when the score has the wrong sign, and from nonnegativity of excess risk when \(R(x)=0\). Taking expectations gives \(E[\delta_\phi(R(X))]\leq S\).

Each calibration function is convex on \([0,1]\). For hinge loss this is immediate. For \(0\leq t<1\), the second derivatives are \(\delta_{\mathrm L}''(t)=1/(1-t^2)>0\) and \(\delta_{\mathrm E}''(t)=1/(1-t^2)^{3/2}>0\); continuity extends convexity to the closed interval. Jensen's inequality therefore yields \(\delta_\phi(E[R(X)])\leq E[\delta_\phi(R(X))]\leq S\). For hinge, \(\delta_{\mathrm H}(R)=R\). For logistic, the series above has nonnegative terms and gives \(\delta_{\mathrm L}(R)\geq R^2/2\), with the endpoint following by continuity. For exponential, \(1-\sqrt{1-R^2}\geq R^2/2\), since \(\sqrt{1-z}\leq1-z/2\) for \(0\leq z\leq1\). These facts give the claimed inequalities. \(\square\)

Consequently, if a sequence of procedures has surrogate excess risk \(S_n=O(n^{-\alpha})\), these bounds give classification regret \(R_n=O(n^{-\alpha})\) for hinge loss and \(R_n=O(n^{-\alpha/2})\) for logistic and exponential loss. This is the rate-exponent distinction caused by linear versus quadratic calibration near zero. These bounds concern conversion from surrogate excess to classification regret; they do not by themselves establish how quickly a particular procedure achieves its surrogate-risk rate.

Common Pitfall

• Confusing the expansion variable. The Taylor expansions are in \(t=|2\eta-1|\), the conditional class imbalance, not in the score \(u\). A quadratic calibration function does not mean the loss itself is quadratic in the score.
• Reading the exponent in the wrong direction. If \(\delta(t)\) is proportional to \(t^2\), then surrogate excess controls regret through a square root. It does not imply that regret is proportional to the square of surrogate excess.
• Treating endpoint infima as finite minimizers. For logistic and exponential losses at \(\eta=1\), the unrestricted infimum is approached as \(u\to+\infty\) and is not attained at a finite score. The calibration formulas remain valid because they use infima.