The Logistic Loss Calibration Function

Why the Conditional Minima Matter

A classification score is useful only if its sign identifies the more probable class. A surrogate loss such as logistic loss does not directly charge zero-one classification errors; instead, it assigns a smooth cost to each score. To quantify what a wrong sign costs, compare two conditional risks: the best risk over all scores and the best risk among scores that have the wrong sign. Their difference is the logistic loss calibration function.

The derivation is pointwise: fix the conditional probability of the positive class, optimize over the real-valued score, and then minimize the resulting risk gap over class probabilities whose Bayes classification advantage is at least a specified amount. No distribution for the feature vector is needed for this calculation.

Conditional Logistic Risk and Calibration

Let \(Y\in\{-1,+1\}\), and write \(\eta=P(Y=+1\mid X=x)\) for a fixed feature value \(x\). A real score \(a\) predicts according to its sign. The logistic margin loss is \(\phi(u)=\log(1+e^{-u})\), with natural logarithms, so the conditional surrogate risk is

The conditional zero-one risks of predicting \(+1\) and \(-1\) are \(1-\eta\) and \(\eta\), respectively. Thus the smaller risk is \(\min(\eta,1-\eta)\), and the Bayes sign is positive when \(\eta>1/2\) and negative when \(\eta<1/2\). The amount by which the Bayes sign improves on the opposite sign is \(|2\eta-1|\).

The central calculation is to find both infima in this definition. We first do so for a fixed \(\eta\), including the boundary cases \(\eta=0\) and \(\eta=1\).

The Unconstrained Conditional-Risk Minimum

For \(0<\eta<1\), differentiating the conditional risk gives

Strict convexity means the unique finite minimizer is the solution of \(R_\eta'(a)=0\), namely \(a_\eta^*=\log\{\eta/(1-\eta)\}\). Substitution gives

At \(\eta=0\), \(R_0(a)=\log(1+e^a)\) has infimum \(0\) as \(a\to-\infty\); at \(\eta=1\), \(R_1(a)=\log(1+e^{-a})\) has infimum \(0\) as \(a\to+\infty\). Neither endpoint infimum is attained at a finite score. Both agree with \(h(\eta)=0\).

The Wrong-Sign Conditional-Risk Minimum

Suppose first that \(\eta>1/2\), so that the Bayes sign is positive. A score with the wrong sign satisfies \(a\leq0\). Since \(R_\eta'(a_\eta^*)=0\), \(a_\eta^*>0\), and \(R_\eta''(a)>0\), the risk is strictly decreasing up to \(a_\eta^*\). It is therefore minimized over \(a\leq0\) at the largest allowed score, \(a=0\), and the constrained minimum is \(R_\eta(0)=\log 2\).

If \(\eta<1/2\), the Bayes sign is negative and the wrong-sign constraint is \(a\geq0\). Now \(a_\eta^*<0\), and strict convexity implies that the risk is increasing to the right of \(a_\eta^*\). Its minimum over \(a\geq0\) is again at \(a=0\), with value \(\log 2\). At \(\eta=1/2\), the unconstrained minimizer is already zero and the minimum is also \(\log 2\).

At \(\eta=0\) and \(\eta=1\), the same constrained statement follows directly: on the wrong-sign half-line the endpoint score \(a=0\) minimizes the risk, while the unconstrained infimum is approached at an infinite score of the Bayes sign.

Deriving the Calibration Function

For a fixed \(\eta\), write \(t=|2\eta-1|\). Binary entropy is symmetric, \(h(\eta)=h(1-\eta)\), and therefore \(h(\eta)=h((1+t)/2)\). The preceding minimization shows that the smallest conditional risk excess under a wrong-sign constraint is \(\log2-h((1+t)/2)\). The remaining step is to minimize this quantity over all \(t\geq\epsilon\).

Proof. Define \(g(t)=\log2-h((1+t)/2)\) for \(0\leq t\leq1\). For \(0<t<1\), differentiating entropy gives

Thus \(g\) is increasing on \((0,1)\), and by continuity it is nondecreasing on the closed interval. Under the calibration constraint \(t\geq\epsilon\), the smallest possible wrong-sign risk excess is consequently \(g(\epsilon)\). It is achievable by taking \(\eta=(1+\epsilon)/2\) and \(a=0\), including the endpoint cases by the definitions of the infima. Hence \(\psi(\epsilon)=g(\epsilon)\), which is the stated formula. For any particular wrong-sign pair \((\eta,a)\), its excess is at least the infimum over all such pairs with \(t=|2\eta-1|\geq t\), namely \(\psi(t)\). \(\square\)

The formula gives \(\psi(0)=\log2-h(1/2)=0\) and \(\psi(1)=\log2-h(1)=\log2\). It is strictly increasing for \(0<\epsilon<1\). The zero value at \(\epsilon=0\) is essential: when the two classes are equally likely, a sign error has no conditional zero-one regret to control.

Why the Formula Is Useful—and a Common Mistake

The calibration function translates conditional classification regret into a lower bound on conditional surrogate-risk excess. If a score has the wrong sign when the Bayes advantage is \(t\), the theorem gives an excess of at least \(\psi(t)\). The formula is exact for the conditional optimization here, rather than merely a convenient bound: the least costly wrong-sign score lies at the boundary \(a=0\).

A common mistake is to use the unconstrained logistic minimum as though it were the wrong-sign minimum. For example, at \(\eta=0.8\), the unconstrained score is \(\log4>0\) and achieves risk \(h(0.8)\); it does not satisfy the wrong-sign constraint. The correct constrained minimization is over \(a\leq0\), whose minimum is \(\log2\). Subtracting the unconstrained minimum from this constrained minimum gives the gap \(\log2-h(0.8)\), not zero.

Another error is to confuse the entropy argument with the classification advantage: when the advantage is \(\epsilon\), the larger class probability is \((1+\epsilon)/2\), not \(\epsilon\). The symmetry \(h(\eta)=h(1-\eta)\) is what allows either class to be handled by the same formula.