Classification Calibration Versus Probability Accuracy

Two Different Goals for a Surrogate

In binary classification, a score can be useful for deciding which class to predict without being useful as an estimate of how likely either class is. This distinction matters whenever a score is interpreted beyond its sign: a clinician may need a risk estimate, for example, while a classifier that only assigns a category may need only the correct side of a decision threshold.

Let $Y\in\{-1,+1\}$, let $X$ be a predictor, and write $\eta(x)=P(Y=+1\mid X=x)$. The Bayes classifier predicts $+1$ when $\eta(x)>1/2$ and $-1$ when $\eta(x)<1/2$; at equality either class has the same conditional zero-one risk. A score $f(x)$ is commonly converted to a class by its sign. Estimating $\eta(x)$, in contrast, requires a score or transformation of the score that carries probability information, not merely the correct sign.

Definition: A margin surrogate has loss $\phi(Yf(X))$, where $f(X)$ is a real-valued score and $\phi$ is a loss function. At a fixed predictor value $x$, its conditional risk is $$ C_\eta(f)=\eta\phi(f)+(1-\eta)\phi(-f),\qquad \eta=\eta(x). $$ A surrogate is classification-calibrated when minimizing this conditional risk selects the Bayes class whenever $\eta\ne 1/2$. This is a statement about the decision sign. Probability estimation is a separate property: a probability loss is strictly proper if its conditional expected loss, viewed as a function of a reported probability $q\in[0,1]$, has the unique minimizer $q=\eta$.

The first property concerns which side of zero a minimizing score lies on. The second concerns whether a probability report equals $\eta$. Classification calibration alone does not require the score to equal $\eta$, nor does it guarantee that applying an arbitrary probability link to the score gives an accurate estimate.

Worked Examples with Hinge Loss

For the hinge loss, $\phi(t)=(1-t)_+=\max(0,1-t)$, the conditional risk is $$ C_\eta(f)=\eta(1-f)_+ +(1-\eta)(1+f)_+. $$ Its minimizers show directly how a surrogate can recover the class while discarding probability information.

Worked Example: Two Probabilities, the Same Hinge Minimizer

Take $\eta=0.7$. For $-1\le f\le 1$, both hinge terms are active, so $$ C_{0.7}(f)=0.7(1-f)+0.3(1+f)=1-0.4f. $$ This decreases as $f$ increases. For $f\ge 1$, the first term is zero and $C_{0.7}(f)=0.3(1+f)$, which increases with $f$. For $f\le -1$, the second term is zero and $C_{0.7}(f)=0.7(1-f)$, which decreases as $f$ increases toward $-1$. Thus the unique minimum is at $f=1$, with risk $0.3(1+1)=0.6$. The score has the correct positive sign, but the raw score $1$ is not the conditional probability $0.7$.

Now take $\eta=0.9$. On $[-1,1]$, $C_{0.9}(f)=1-0.8f$, so it decreases up to $f=1$. For $f\ge1$, it equals $0.1(1+f)$, which increases; for $f\le-1$, it equals $0.9(1-f)$, which decreases toward $-1$. The unique minimum is again $f=1$, and its risk is $0.1(1+1)=0.2$. The two different probabilities $0.7$ and $0.9$ therefore have the same minimizing score. No fixed transformation of that minimizing score can recover both probabilities.

The endpoint probabilities need separate attention because the hinge risk can be flat beyond its margin. The full minimizer description is as follows.

Result: For hinge loss, if $1/2<\eta<1$, the unique conditional-risk minimizer is $f=1$; if $\eta=1$, every $f\ge1$ minimizes the risk. If $0<\eta<1/2$, the unique minimizer is $f=-1$; if $\eta=0$, every $f\le-1$ minimizes it. At $\eta=1/2$, every $f\in[-1,1]$ minimizes it.

To verify the result, on $[-1,1]$ the risk is $1+(1-2\eta)f$. For $f\le-1$, it is $\eta(1-f)$, and for $f\ge1$, it is $(1-\eta)(1+f)$. When $1/2<\eta<1$, the middle piece decreases, the left piece decreases toward $-1$, and the right piece increases from $1$; the minimum is uniquely at $1$. When $0<\eta<1/2$, the middle piece increases, the left piece decreases toward $-1$, and the right piece increases from $1$; the minimum is uniquely at $-1$. At $\eta=1$, the risk is $(1-f)_+$, which is zero for every $f\ge1$; at $\eta=0$, it is $(1+f)_+$, which is zero for every $f\le-1$. At $\eta=1/2$, the middle piece is constantly $1$, while the two outer pieces are at least $1$, proving the stated set of minimizers. $\square$

What Hinge Calibration Guarantees

Although hinge minimizers do not identify $\eta$, hinge-risk minimization does control classification regret. Use the convention $g_f(x)=+1$ if $f(x)>0$ and $g_f(x)=-1$ otherwise. At a fixed $x$, predicting the wrong class when $\eta\ne1/2$ incurs zero-one risk regret $|2\eta-1|$: the conditional risks of predicting $+1$ and $-1$ are $1-\eta$ and $\eta$, respectively.

Theorem: For every $\eta\in[0,1]$ and every score $f$, hinge conditional-risk excess is at least conditional zero-one regret: $$ C_\eta(f)-\inf_{t\in\mathbb R}C_\eta(t) \ \ge\ |2\eta-1|\,\mathbf{1}\{g_f\ne g^*\}, $$ where the right side is zero at $\eta=1/2$, and $g^*$ is a Bayes class choice.

Proof. First suppose $\eta>1/2$, so the Bayes class is $+1$. A wrong prediction under the stated convention has $f\le0$. If $-1\le f\le0$, then $C_\eta(f)=1+(1-2\eta)f\ge1$, because $1-2\eta<0$ and $f\le0$. If $f<-1$, then $C_\eta(f)=\eta(1-f)>2\eta>1$. The minimum hinge risk is $2(1-\eta)$, attained at $f=1$ for interior $\eta$, and also given by this expression at $\eta=1$. Thus for any wrong-sign score, $$ C_\eta(f)-\inf_t C_\eta(t)\ge 1-2(1-\eta)=2\eta-1. $$ Now suppose $\eta<1/2$, so the Bayes class is $-1$. A wrong prediction has $f>0$. For $0<f\le1$, $C_\eta(f)=1+(1-2\eta)f\ge1$. For $f>1$, $C_\eta(f)=(1-\eta)(1+f)>2(1-\eta)>1$. The minimum risk is $2\eta$, including at $\eta=0$, so the excess is at least $1-2\eta$. At $\eta=1/2$, the claimed lower bound is zero and the excess is nonnegative by the definition of an infimum. These cases prove the inequality. $\square$

Integrating the inequality over $X$ shows that population hinge excess risk bounds population zero-one regret. Consequently, if a sequence of predictors has hinge excess risk tending to zero, its classification regret also tends to zero. The bound does not say that the scores estimate $\eta(X)$: in the worked examples, hinge minimizers for distinct conditional probabilities coincide. Small classification regret and accurate probability estimation answer different questions.

Logistic and Squared-Hinge Scores

Other surrogates illustrate a complementary point: even when a score contains enough information to recover $\eta$, the raw score need not itself be a probability. The appropriate interpretation depends on the loss and its associated link.

Worked Example: Logistic Loss and Its Probability Link

For logistic loss, $\phi(t)=\log(1+e^{-t})$, the conditional risk is $$ C_\eta(f)=\eta\log(1+e^{-f})+(1-\eta)\log(1+e^f). $$ For $0<\eta<1$, differentiating gives $$ C_\eta'(f)=\frac{(1-\eta)e^f-\eta}{1+e^f}. $$ The derivative is zero exactly when $e^f=\eta/(1-\eta)$, so the unique minimizer is $f^*=\log\{\eta/(1-\eta)\}$. The second derivative is $e^f/(1+e^f)^2>0$, confirming a strict minimum. The score is the log-odds, not the probability. Applying the logistic link $\sigma(f)=1/(1+e^{-f})$ gives $\sigma(f^*)=\eta$. For example, when $\eta=0.8$, $f^*=\log 4$, and $\sigma(\log4)=4/(1+4)=0.8$. The minimum conditional risk is $-0.8\log(0.8)-0.2\log(0.2)$, approximately $0.5004$.

At $\eta=0$ or $\eta=1$, the logistic risk has infimum zero as $f$ tends to $-\infty$ or $+\infty$, respectively; no finite score attains that infimum. The probability statement at these endpoints is understood through the limiting linked probability.

This probability interpretation follows from the conditional log loss. If the reported probability is $q=\sigma(f)$, the conditional risk can be written as $-\eta\log q-(1-\eta)\log(1-q)$. For $0<q<1$, its derivative with respect to $q$ is $(q-\eta)/(q(1-q))$. It is negative for $q<\eta$ and positive for $q>\eta$, so the unique minimum is at $q=\eta$. Logistic loss is therefore strictly proper after its probability link, while its untransformed score is on the log-odds scale.

Worked Example: A Squared-Hinge Score Is Not a Probability

Take squared-hinge loss $\phi(t)=(1-t)_+^2$ and $\eta=0.75$. For $-1\le f\le1$, its conditional risk is $$ C_{0.75}(f)=0.75(1-f)^2+0.25(1+f)^2. $$ Differentiating gives $C_{0.75}'(f)=-1.5(1-f)+0.5(1+f)=-1+2f$, so the minimum on this interval is at $f=0.5$. The risk there is $0.75(0.5)^2+0.25(1.5)^2=0.1875+0.5625=0.75$. For $f>1$, the risk is $0.25(1+f)^2$, which is increasing and has value $1$ at $f=1$. For $f<-1$, it is $0.75(1-f)^2$, which decreases as $f$ increases toward $-1$ and has value $3$ at $-1$. Thus $f=0.5$ is the global minimizer. It has the correct positive sign, but its raw value $0.5$ differs from $\eta=0.75$.

For interior $\eta$, the same calculation on $[-1,1]$ gives the minimizer $f^*=2\eta-1$. Thus the link $q=(f+1)/2$ recovers $\eta$ at the conditional minimizer, whereas treating $f$ itself as a probability does not. In this example, the linked estimate is $(0.5+1)/2=0.75$.

Common Mistake: Treating a Correct Sign as a Calibrated Probability

A common error is to conclude that because a loss is classification-calibrated, its minimizing scores must be close to $\eta$. The hinge example disproves that inference: $\eta=0.7$ and $\eta=0.9$ both have the unique minimizing score $f=1$. The guarantee concerns the decision boundary at $1/2$, not the numerical scale of scores or their distance from a probability.

The correction is to identify the intended output before interpreting a score. For classification, check whether the sign selects the Bayes class and whether excess surrogate risk controls classification regret. For probability estimation, specify a probability report $q$, the loss used to assess it, and the link from score to probability; then check that the conditional expected loss is minimized at $q=\eta$. A probability link cannot restore information that a score has discarded, as the two hinge minimizers demonstrate.

Key takeaway: Classification calibration is about selecting the right class. Probability estimation is about recovering the conditional class probability. Some surrogates provide both properties with an appropriate link; classification calibration by itself guarantees only the first.

Check Your Understanding

Use the conditional risks and definitions in this tutorial to answer the following.

For hinge loss with $\eta=0.6$, find the conditional-risk minimizer and its minimum risk. Is the minimizing score a probability estimate?
Why can no fixed transformation of the hinge conditional minimizer recover both $\eta=0.7$ and $\eta=0.9$?
For logistic loss, what is the minimizing score when $\eta=0.25$, and what probability does the logistic link return?
State the hinge conditional excess-risk lower bound for a score that predicts the wrong class when $\eta=0.8$.
For squared-hinge loss at an interior probability $\eta$, what score minimizes the conditional risk, and which linear link recovers $\eta$?