Margin Scores and Conditional Surrogate Risk

Scores, Labels, and Margins

A binary classifier can report a real-valued score before converting that score into a class decision. The score carries more information than its sign: its magnitude indicates how far it lies from the decision threshold, while its sign selects a class. A margin loss evaluates the score after orienting it by the observed label. This makes it possible to express the expected loss at a feature value using only the conditional probability of the positive class.

Throughout, the label $Y$ takes values in $\{-1,+1\}$, and $s\in\mathbb{R}$ is a score. A positive score favors label $+1$; a negative score favors label $-1$. We use the decision convention that a score of zero predicts $+1$. That convention matters when the conditional probability of $+1$ is less than one half.

Definition: A margin loss is a measurable function $\phi:\mathbb{R}\to[0,\infty]$ used to assign loss $\phi(ys)$ to label $y\in\{-1,+1\}$ and score $s$. For $\eta=P(Y=+1\mid X=x)$, the conditional surrogate risk at score $s$ is $$ C_{\eta}(s)=\eta\phi(s)+(1-\eta)\phi(-s). $$ The two terms are the loss conditional on $Y=+1$ and on $Y=-1$, weighted by their conditional probabilities.

The formula follows because $ys=s$ when $y=+1$, and $ys=-s$ when $y=-1$. It applies whether the loss is smooth, piecewise linear, or discontinuous. If a risk comparison subtracts two conditional risks, assume the quantities being compared are finite; otherwise the difference might not be defined.

Classification Risk and Score Sign

Let $h_s=+1$ if $s\geq 0$, and $h_s=-1$ if $s<0$. Conditional on $X=x$, predicting $+1$ has zero-one risk $1-\eta$, while predicting $-1$ has risk $\eta$. The Bayes decision is therefore $+1$ when $\eta>1/2$, $-1$ when $\eta<1/2$, and either decision when $\eta=1/2$. Its conditional risk is $\min\{\eta,1-\eta\}$.

Theorem: Under the stated zero-score convention, the conditional excess zero-one risk of score $s$ is $$ \Delta_{01}(\eta,s) = |2\eta-1| \left[ \mathbf{1}\{\eta>1/2,\ s<0\} + \mathbf{1}\{\eta<1/2,\ s\geq 0\} \right]. $$ Thus a sign error costs $|2\eta-1|$, and a correct Bayes decision costs zero. At $\eta=1/2$, every decision has zero excess risk.

Proof. If $\eta>1/2$, the Bayes decision is $+1$, with risk $1-\eta$. A score predicts $-1$ exactly when $s<0$; that prediction has risk $\eta$, so its excess risk is $\eta-(1-\eta)=2\eta-1$. If the score predicts $+1$, its risk equals the Bayes risk and the excess is zero.

If $\eta<1/2$, the Bayes decision is $-1$, with risk $\eta$. A score predicts $+1$ exactly when $s\geq0$; that prediction has risk $1-\eta$, so its excess risk is $(1-\eta)-\eta=1-2\eta$. A negative score predicts $-1$ and has zero excess risk. Finally, when $\eta=1/2$, both decisions have risk $1/2$. These cases give the stated expression. $\square$

The result separates two ingredients: whether the score’s sign chooses the wrong class, and how far $\eta$ is from the decision boundary. A wrong decision near $\eta=1/2$ has small conditional regret; the same error when $\eta$ is far from one half has greater regret. The zero-score convention appears in the condition $s\geq0$: for $\eta<1/2$, zero is a wrong decision.

Worked Conditional-Risk Calculations

Worked Example: A Hinge Margin Loss

Suppose $P(Y=+1\mid X=x)=\eta=0.8$, and use the hinge margin loss $\phi(u)=\max\{0,1-u\}$. Calculate the conditional surrogate risks at scores $s=-0.5$ and $s=0.5$, then find the classification regret for the negative score.

At $s=-0.5$, the two margin losses are $\phi(-0.5)=1.5$ and $\phi(0.5)=0.5$. Therefore $C_{0.8}(-0.5)=0.8(1.5)+0.2(0.5)=1.2+0.1=1.3$.

At the reflected score $s=0.5$, they are reversed: $C_{0.8}(0.5)=0.8(0.5)+0.2(1.5)=0.4+0.3=0.7$. The negative score predicts $-1$, although $\eta>1/2$ makes $+1$ the Bayes decision. Its classification regret is $0.8-0.2=0.6=|2(0.8)-1|$. Here reflecting the score lowers conditional surrogate risk by $1.3-0.7=0.6$.

Worked Example: A Logistic Margin Loss

Let $\eta=0.25$ and $\phi(u)=\log(1+\exp(-u))$. Compare scores $1$ and $-1$, and calculate the classification regret of score $1$.

The loss values are $\phi(1)=\log(1+e^{-1})\approx0.3133$ and $\phi(-1)=\log(1+e)\approx1.3133$. Thus $C_{0.25}(1)=0.25(0.3133)+0.75(1.3133)\approx0.0783+0.9850=1.0633$.

For the opposite score, $C_{0.25}(-1)=0.25(1.3133)+0.75(0.3133)\approx0.3283+0.2350=0.5633$. Because $\eta<1/2$, the Bayes decision is $-1$. Score $1$ predicts $+1$, so its risk is $1-\eta=0.75$, compared with Bayes risk $0.25$. The regret is $0.75-0.25=0.50=|2(0.25)-1|$.

Worked Example: A Squared Margin Loss

Take $\eta=0.6$, $\phi(u)=(1-u)^2$, and scores $-1$ and $1$. Compute both conditional surrogate risks and the regret for score $-1$.

The loss values are $\phi(-1)=4$ and $\phi(1)=0$. Hence $C_{0.6}(-1)=0.6(4)+0.4(0)=2.4$, while $C_{0.6}(1)=0.6(0)+0.4(4)=1.6$.

Since $\eta>1/2$, the Bayes decision is $+1$, with risk $0.4$. Score $-1$ predicts $-1$, with risk $0.6$, so its regret is $0.6-0.4=0.2=|2(0.6)-1|$. The conditional surrogate risks differ by $2.4-1.6=0.8$; this is a loss-scale comparison, not the zero-one regret.

What Reflection of the Score Can Tell Us

For a fixed score magnitude, comparing $s$ with $-s$ isolates the effect of reversing its sign. The identity below is purely algebraic. Its implication for which sign is preferred needs an additional shape condition on the loss.

Theorem: For any $\eta\in[0,1]$ and finite values $\phi(s)$ and $\phi(-s)$, $$ C_{\eta}(s)-C_{\eta}(-s) = (2\eta-1)\bigl(\phi(s)-\phi(-s)\bigr). $$ If, in addition, $\phi$ is nonincreasing, then reflecting a nonzero score that chooses the wrong class cannot increase conditional surrogate risk.

Proof. By definition, $C_{\eta}(-s)=\eta\phi(-s)+(1-\eta)\phi(s)$. Subtract this from $C_{\eta}(s)=\eta\phi(s)+(1-\eta)\phi(-s)$. Collecting terms gives $(\eta-(1-\eta))\phi(s)+((1-\eta)-\eta)\phi(-s)$, which equals the displayed identity.

For the implication, first suppose $\eta>1/2$ and the score is wrong, so $s<0$. Then $s<-s$, and nonincreasing $\phi$ gives $\phi(s)\geq\phi(-s)$. Both factors on the right-hand side of the identity are nonnegative, so $C_{\eta}(s)\geq C_{\eta}(-s)$. Now suppose $\eta<1/2$ and $s>0$ is wrong. Nonincreasing $\phi$ gives $\phi(s)\leq\phi(-s)$; both factors are nonpositive, so their product is again nonnegative. In either case the reflected score has no larger conditional surrogate risk. If $s=0$, reflection changes nothing. $\square$

This comparison does not say that surrogate-risk differences equal classification regrets. In the first worked example the two differences happen to be equal; in the squared-margin example they are $0.8$ and $0.2$. Their scales depend on $\phi$. Nor does nonincreasing behavior alone guarantee a strict preference: a loss can be flat over the two margins being compared.

From Conditional to Population Risk

When a measurable score function $f(X)$ is applied to random features, the conditional expression also gives the population surrogate risk. Define $R_{\phi}(f)=E[\phi(Yf(X))]$ and let $\eta(X)=E[\mathbf{1}\{Y=+1\}\mid X]$, a version of the conditional class probability. For measurable nonnegative $\phi$, conditioning on $X$ gives

R_{\phi}(f) = E\!\left[ \eta(X)\phi(f(X))+ \bigl(1-\eta(X)\bigr)\phi(-f(X)) \right] = E[C_{\eta(X)}(f(X))].

This equality follows from the defining property of conditional expectation: given $X$, the score $f(X)$ is fixed, and $Y$ takes the values $+1$ and $-1$ with conditional probabilities $\eta(X)$ and $1-\eta(X)$. Nonnegativity permits the conditional-expectation identity even when a risk is infinite. The analogous population zero-one regret is the expectation of the pointwise regret derived above. Thus the conditional calculation identifies exactly where classification mistakes occur and how much each contributes.

Common Pitfall

A frequent error is to treat a low conditional surrogate risk as if it directly specified whether the score is on the correct side of zero, or to equate a surrogate-risk difference with a zero-one regret. Neither conclusion follows from the definition alone. For instance, with the constant margin loss $\phi(u)=1$, every score has $C_{\eta}(s)=1$, even when its sign predicts the wrong class and its classification regret is positive. The correction is to calculate the decision and its regret from $\eta$ and the score sign, then analyze surrogate risk using the particular loss and any conditions imposed on it. A sign comparison is not, by itself, a quantitative regret bound.

Key takeaway: The conditional surrogate risk is the probability-weighted average of the two label-specific margin losses. The conditional classification regret, by contrast, is $|2\eta-1|$ precisely when the score-based decision disagrees with a Bayes decision. Keep these quantities distinct.

Check Your Understanding

Use the zero-score convention and the definitions above.

For $\eta=0.7$ and $s=-0.2$, what is the conditional excess zero-one risk? Identify the Bayes decision.
Write $C_{\eta}(s)$ explicitly when $\eta=0.3$, in terms of $\phi(s)$ and $\phi(-s)$.
When $\eta=1/2$, can either score sign have positive conditional excess zero-one risk? Explain.
For a nonincreasing margin loss, why does reflecting a wrong score $s<0$ when $\eta>1/2$ not increase conditional surrogate risk?
Why does a comparison of two conditional surrogate risks not generally equal the difference in their zero-one risks?