Score Minimizers, Links, and Sign Recovery

Conditional Scores and Their Meaning

A margin score is useful only if its value leads to an appropriate classification decision. In binary classification, write the label as $Y\in\{-1,+1\}$, and let $t\in\mathbb{R}$ be a score: its sign predicts a class, while its magnitude indicates how far the score lies from the decision boundary at zero. This tutorial examines what score is preferred by two standard surrogate losses when the conditional class probability is fixed.

Fix a covariate value $x$ at which the conditional probability is defined, and write $\eta=P(Y=+1\mid X=x)$. The conditional probability of the negative class is $1-\eta$. For a margin loss $\phi$, a positive label contributes loss $\phi(t)$, and a negative label contributes loss $\phi(-t)$. Averaging over the conditional label distribution gives the conditional surrogate risk.

Definition: For a margin loss $\phi:\mathbb{R}\to[0,\infty)$, the conditional risk at class probability $\eta$ and score $t$ is $$ C_\phi(\eta,t)=\eta\phi(t)+(1-\eta)\phi(-t). $$ A conditional score minimizer is a value of $t$ minimizing $C_\phi(\eta,t)$ over $t\in\mathbb{R}$. The Bayes class is $+1$ when $\eta>1/2$ and $-1$ when $\eta<1/2$.

The target here is pointwise: for each fixed $\eta$, minimize the one-variable function $C_\phi(\eta,t)$. The formulas will show both the score scale and the way that scale relates to the class probability. We first treat $0<\eta<1$, which ensures that both classes have positive conditional probability and that the minimizers are finite.

Logistic Loss: Log-Odds as the Optimal Score

The logistic margin loss is $\phi_{\mathrm{log}}(t)=\log(1+e^{-t})$. Its conditional risk is

C_{\mathrm{log}}(\eta,t) =\eta\log(1+e^{-t})+(1-\eta)\log(1+e^t).

Differentiate with respect to the score. The derivative of the first loss term includes a minus sign from differentiating $-t$; after putting the two terms over a common denominator, it is

\frac{\partial}{\partial t}C_{\mathrm{log}}(\eta,t) =-\frac{\eta}{1+e^t}+\frac{(1-\eta)e^t}{1+e^t} =\frac{(1-\eta)e^t-\eta}{1+e^t}.

The denominator is strictly positive. Thus the derivative vanishes exactly when $(1-\eta)e^t=\eta$, or $t=\log(\eta/(1-\eta))$. To establish that this stationary point is the global minimizer, not merely a candidate, differentiate once more:

\frac{\partial^2}{\partial t^2}C_{\mathrm{log}}(\eta,t) =\frac{e^t}{(1+e^t)^2}>0.

Theorem: If $0<\eta<1$, the logistic conditional risk has the unique global minimizer $$ t_{\mathrm{log}}^*(\eta)=\log\!\left(\frac{\eta}{1-\eta}\right). $$ Proof: The displayed second derivative is positive for every finite $t$, so the conditional risk is strictly convex. Its first derivative is zero exactly at $t=\log(\eta/(1-\eta))$. A stationary point of a differentiable strictly convex function is its unique global minimizer. $\square$

This formula is the log-odds of the positive class. Equivalently, applying the logistic probability link $q_{\mathrm{log}}(t)=1/(1+e^{-t})$ to the optimal score recovers the conditional probability:

q_{\mathrm{log}}\bigl(t_{\mathrm{log}}^*(\eta)\bigr) =\frac{1}{1+\exp\{-\log(\eta/(1-\eta))\}} =\eta.

Thus the minimizing logistic score is not itself a probability: it is the log-odds, which can be any real number. The probability is obtained by applying the increasing link $q_{\mathrm{log}}$.

Worked Example: Logistic Minimization at a Three-to-One Odds Ratio

Suppose $P(Y=+1\mid X=x)=3/4$, so $\eta=3/4$ and $1-\eta=1/4$. The odds are $\eta/(1-\eta)=3$, and the candidate minimizer is $t^*_{\mathrm{log}}=\log 3$.

Substitution in the derivative numerator gives $(1-\eta)e^{t^*_{\mathrm{log}}}-\eta=(1/4)3-3/4=0$. The second derivative is positive at every finite score, so this stationary point is the unique global minimum. Its score is positive, consistent with the Bayes class $+1$.

The logistic link recovers the probability: $q_{\mathrm{log}}(\log 3)=1/(1+e^{-\log 3})=1/(1+1/3)=3/4$. The minimum risk can also be checked directly: $C_{\mathrm{log}}(3/4,\log 3)=(3/4)\log(4/3)+(1/4)\log 4$.

Exponential Loss: Half the Log-Odds

For the exponential margin loss, $\phi_{\mathrm{exp}}(t)=e^{-t}$, the conditional risk is

C_{\mathrm{exp}}(\eta,t)=\eta e^{-t}+(1-\eta)e^t.

Its first and second derivatives are

\frac{\partial}{\partial t}C_{\mathrm{exp}}(\eta,t) =-\eta e^{-t}+(1-\eta)e^t, \qquad \frac{\partial^2}{\partial t^2}C_{\mathrm{exp}}(\eta,t) =\eta e^{-t}+(1-\eta)e^t.

For $0<\eta<1$, the second derivative is positive at every finite $t$. Setting the first derivative to zero gives $(1-\eta)e^t=\eta e^{-t}$, hence $e^{2t}=\eta/(1-\eta)$. Taking logarithms yields a score that is half the logistic log-odds.

Theorem: If $0<\eta<1$, the exponential conditional risk has the unique global minimizer $$ t_{\mathrm{exp}}^*(\eta)=\frac{1}{2}\log\!\left(\frac{\eta}{1-\eta}\right). $$ Proof: The second derivative is strictly positive for every finite $t$, so the risk is strictly convex. The first derivative vanishes precisely when $e^{2t}=\eta/(1-\eta)$, which has the unique real solution displayed. Strict convexity makes that stationary point the unique global minimizer. $\square$

The corresponding probability link must account for the factor of one half. Define $q_{\mathrm{exp}}(t)=1/(1+e^{-2t})$. Then

q_{\mathrm{exp}}\bigl(t_{\mathrm{exp}}^*(\eta)\bigr) =\frac{1}{1+\exp\{-\log(\eta/(1-\eta))\}} =\eta.

Both losses therefore encode the same conditional probability at their respective minimizing scores, but their score scales differ: the exponential minimizer is half the logistic minimizer. Using the logistic link with an exponential score, or the exponential link with a logistic score, would generally give the wrong probability.

Worked Example: Exponential Minimization at a Three-to-One Odds Ratio

Again take $\eta=3/4$. The exponential stationary equation is $-(3/4)e^{-t}+(1/4)e^t=0$. Multiplying by $e^t>0$ gives $e^{2t}=3$, so $t^*_{\mathrm{exp}}=(1/2)\log 3$, which is positive.

The second derivative is positive, so this is the unique global minimizer. Its exponential link gives $q_{\mathrm{exp}}((1/2)\log 3)=1/(1+e^{-\log 3})=3/4$.

The minimum risk is $(3/4)e^{-(1/2)\log 3}+(1/4)e^{(1/2)\log 3}$. Since $e^{(1/2)\log 3}=\sqrt{3}$, this equals $(3/4)(1/\sqrt{3})+(1/4)\sqrt{3}=\sqrt{3}/2$. The score is half the logistic score from the preceding example, even though both links return the same probability.

Why Both Minimizers Recover the Bayes Sign

For $0<\eta<1$, both formulas use the log-odds $\log(\eta/(1-\eta))$, multiplied by a positive constant: one for logistic loss and one half for exponential loss. The logarithm is strictly increasing, so the sign of this quantity is determined by whether its argument is greater or less than one.

Result: If $0<\eta<1$ and $\eta\ne 1/2$, then $$ \operatorname{sign}\bigl(t_{\mathrm{log}}^*(\eta)\bigr) =\operatorname{sign}\bigl(t_{\mathrm{exp}}^*(\eta)\bigr) =\operatorname{sign}(\eta-1/2). $$ In particular, each conditional-risk minimizer has the Bayes class sign.

To verify the sign directly, $\eta/(1-\eta)>1$ exactly when $\eta>1-\eta$, which is equivalent to $\eta>1/2$. Its logarithm is then positive. If $\eta<1/2$, the ratio is less than one and its logarithm is negative. Multiplication by $1/2$ does not change a sign. This establishes sign recovery without equating score magnitude across losses.

Worked Example: A Negative Bayes Class and Different Score Scales

Suppose $\eta=1/5$, so $1-\eta=4/5$. The odds are $1/4$, and the log-odds are $\log(1/4)=-\log 4$. The logistic minimizer is therefore $-\log 4$, while the exponential minimizer is $-\tfrac12\log 4=-\log 2$. Both are negative, as required for the Bayes class $-1$.

For logistic loss, the derivative numerator at $t=-\log 4$ is $(4/5)(1/4)-1/5=1/5-1/5=0$. For exponential loss, the derivative is $-(1/5)e^{\log 2}+(4/5)e^{-\log 2}=-2/5+2/5=0$. The strict convexity established above makes each stationary score the unique global minimizer.

The two probability links agree at their respective scores: $q_{\mathrm{log}}(-\log 4)=1/(1+4)=1/5$, and $q_{\mathrm{exp}}(-\log 2)=1/(1+e^{2\log 2})=1/(1+4)=1/5$. The scores differ because their link functions use different scales, not because they represent different conditional probabilities.

Endpoint Probabilities and a Common Pitfall

The formulas for finite minimizers assume $0<\eta<1$. At an endpoint, the conditional risk may have an infimum without attaining it at any real score. For example, if $\eta=1$, then $C_{\mathrm{log}}(1,t)=\log(1+e^{-t})$ and $C_{\mathrm{exp}}(1,t)=e^{-t}$. Both decrease toward zero as $t\to+\infty$, but remain positive for every finite $t$. Thus their infimum is zero and neither has a finite minimizer. If $\eta=0$, the corresponding risks decrease toward zero as $t\to-\infty$, again without a finite minimizer. These cases do not contradict the interior formulas; the log-odds diverge at the endpoints.

Worked Example: Why an Endpoint Is Not a Finite Minimizer

Let $\eta=1$. For exponential loss, $C_{\mathrm{exp}}(1,t)=e^{-t}$. At $t=5$, the risk is $e^{-5}>0$, and at any larger finite score it is smaller but still positive. Its infimum is zero because $e^{-t}\to0$ as $t\to+\infty$, but no finite $t$ gives risk zero. For logistic loss, $C_{\mathrm{log}}(1,t)=\log(1+e^{-t})$, which likewise tends to zero as $t\to+\infty$ and is positive for every finite $t$.

A common mistake is to say that the exponential minimizer equals the logistic minimizer because both are described by the log-odds, or to use the same probability link for both scores. The actual formulas are $t_{\mathrm{log}}^*=\log(\eta/(1-\eta))$ and $t_{\mathrm{exp}}^*=\tfrac12\log(\eta/(1-\eta))$. The correction is to keep the loss-specific score scale and its matching link together. Another mistake is to infer a finite minimizer at $\eta=0$ or $1$ by substituting into the interior formula; the divergence of the log-odds signals that the finite-minimizer result no longer applies.

These calculations explain a useful distinction: a score is not automatically a probability, but under the relevant conditional-risk minimization its loss-specific link returns the conditional class probability. For classification alone, the positive rescaling between the two optimal scores does not change the sign when $\eta\ne1/2$. Probability interpretation, however, requires knowing which loss and link produced the score.

Check Your Understanding

Use the conditional-risk formulas and the assumptions stated above to answer the following.

For logistic loss and $0<\eta<1$, differentiate the conditional risk and solve for its unique minimizer.
Why is the exponential score minimizer half the logistic score minimizer, and which probability link recovers $\eta$ from an exponential score?
If $\eta=2/3$, determine the signs of both optimal scores and explain why they agree with the Bayes class.
At $\eta=0$, why is the exponential-risk infimum not attained by any finite score?
Explain why applying the logistic probability link to the exponential minimizer generally does not recover $\eta$.