Scaling Margin Losses and Calibration Constants

Why Scaling a Loss Requires Care

A margin loss evaluates a score through the product of the label and the score. Replacing a loss $\phi(u)$ by $\phi(cu)$, for $c>0$, changes the numerical scale of that margin. For unrestricted scores, this change can be exactly offset by measuring scores in different units. But a penalty or a restricted score class can make that offset unavailable, or make it costly.

The distinction matters when comparing calibration guarantees. The calibration function describes the least surrogate regret required by a given amount of classification regret. It is not, in general, a statement that two learning procedures with different penalties or score classes have the same fitted scores.

Conditional Risk and the Calibration Function

Consider binary labels $Y\in\{-1,+1\}$, with $P(Y=+1\mid X=x)=\eta$. For a score $\alpha\in\mathbb R$, the conditional margin risk under a loss $\phi$ is

C^\phi_\eta(\alpha)=\eta\phi(\alpha)+(1-\eta)\phi(-\alpha).

The scaled loss is $\phi_c(u)=\phi(cu)$, where $c>0$. Its conditional risk is $C^{\phi_c}_\eta(\alpha)$. We use the sign of $\alpha$ as the predicted class, and treat a score of zero as a boundary tie. Thus, a score is wrong or tied when its sign does not agree strictly with the conditional majority class.

Definition: The unrestricted conditional minimum is $H^\phi_\eta=\inf_{\alpha\in\mathbb R} C^\phi_\eta(\alpha)$. For $0\leq\varepsilon\leq1$, define the binary calibration function $$ \delta_\phi(\varepsilon)= \inf_{\substack{\eta\in[0,1]\\ |2\eta-1|\geq\varepsilon}} \ \inf_{\substack{\alpha\in\mathbb R\\ \alpha(2\eta-1)\leq0}} \left\{C^\phi_\eta(\alpha)-H^\phi_\eta\right\}. $$ Here $|2\eta-1|$ is the conditional classification-regret margin, and the inner constraint restricts attention to wrong or tied scores. The infima need not be attained.

This definition concerns unrestricted real-valued scores. It measures a pointwise tradeoff: among conditional distributions with margin at least $\varepsilon$, how small can the surrogate excess risk be while the score is wrong or tied? The scale transformation has a particularly clean effect on this quantity.

What Positive Scaling Changes—and What It Does Not

Theorem: Suppose $\phi:\mathbb R\to[0,\infty)$ is finite-valued and $c>0$. Then $$ H^{\phi_c}_\eta=H^\phi_\eta,\qquad \operatorname*{argmin}_{\alpha\in\mathbb R}C^{\phi_c}_\eta(\alpha) =\frac{1}{c}\operatorname*{argmin}_{t\in\mathbb R}C^\phi_\eta(t), \qquad \delta_{\phi_c}(\varepsilon)=\delta_\phi(\varepsilon). $$ The argmin identity is an identity of sets; it applies also when the sets are empty.

Proof. For every $\alpha$, direct substitution gives

C^{\phi_c}_\eta(\alpha) =\eta\phi(c\alpha)+(1-\eta)\phi(-c\alpha) =C^\phi_\eta(c\alpha).

The map $\alpha\mapsto t=c\alpha$ is a bijection from $\mathbb R$ to $\mathbb R$. Taking the infimum over $\alpha$ therefore gives $H^{\phi_c}_\eta=H^\phi_\eta$. The same identity shows that $\alpha$ minimizes the scaled risk exactly when $c\alpha$ minimizes the original risk, proving the argmin relation.

For the calibration function, use the same substitution in its inner infimum. Since $c>0$, the constraint is preserved: $\alpha(2\eta-1)\leq0$ if and only if $(c\alpha)(2\eta-1)\leq0$. The objective excess is also preserved:

C^{\phi_c}_\eta(\alpha)-H^{\phi_c}_\eta =C^\phi_\eta(c\alpha)-H^\phi_\eta.

Thus the change of variable gives equal inner infima for every $\eta$; taking the outer infimum over the same set of $\eta$'s proves $\delta_{\phi_c}(\varepsilon)=\delta_\phi(\varepsilon)$. No convexity or differentiability assumption is needed for this scaling identity. $\square$

The theorem separates score units from classification behavior. Unrestricted scores can shrink by the factor $1/c$, preserving their signs and conditional excess risks. The scale of a score minimizer changes, but the best conditional risk and the calibration function do not.

Worked Example: Logistic Loss and the Rescaled Score

Take $\phi(u)=\log(1+e^{-u})$, and let $\eta=0.8$. The conditional risk is

C^\phi_\eta(\alpha) =0.8\log(1+e^{-\alpha})+0.2\log(1+e^\alpha).

Differentiating with respect to $\alpha$ gives

\frac{d}{d\alpha}C^\phi_\eta(\alpha) =-\frac{\eta}{1+e^\alpha} +\frac{1-\eta}{1+e^{-\alpha}} =\frac{1}{1+e^{-\alpha}}-\eta.

The derivative is zero when $1/(1+e^{-\alpha})=\eta$, so the unique minimizer is $\alpha^*=\log(\eta/(1-\eta))=\log 4\approx1.386$. Under scaling by $c=2$, the minimizer is $\alpha^*/2\approx0.693$. Its sign remains positive, and the theorem shows that its conditional minimum risk is exactly the same as before scaling.

Two Calibration Functions Under Scaling

The invariance theorem can be checked directly for familiar losses. These calculations also illustrate that the calibration function records excess risk, not the numerical size of the score.

Worked Example: Logistic Calibration Function

For logistic loss, write $\sigma(\alpha)=1/(1+e^{-\alpha})$. The conditional risk has derivative $\sigma(\alpha)-\eta$, so its unrestricted minimum occurs at $\alpha=\log(\eta/(1-\eta))$ when $0<\eta<1$. Substitution gives

H^\phi_\eta=-\eta\log\eta-(1-\eta)\log(1-\eta).

If $\eta>1/2$, a wrong or tied score satisfies $\alpha\leq0$. On this half-line, $\sigma(\alpha)\leq1/2<\eta$, so the conditional risk is decreasing as $\alpha$ increases. Its minimum over $\alpha\leq0$ is therefore at $\alpha=0$, where the risk is $\log 2$. For $\eta<1/2$, symmetry gives the same result with $\eta$ replaced by $1-\eta$. Consequently, at conditional margin $m=|2\eta-1|$, the least wrong-or-tied excess is

\log 2+ \frac{1+m}{2}\log\frac{1+m}{2} +\frac{1-m}{2}\log\frac{1-m}{2}.

This expression increases with $m$ on $[0,1]$: for $0<m<1$, its derivative is $\tfrac12\log((1+m)/(1-m))>0$. Hence the infimum over $m\geq\varepsilon$ occurs at $m=\varepsilon$, and $\delta_\phi(\varepsilon)=\log2-h((1+\varepsilon)/2)$, where $h(p)=-p\log p-(1-p)\log(1-p)$. Positive scaling leaves this entire function unchanged.

Worked Example: Exponential Calibration Function

For $\phi(u)=e^{-u}$, the conditional risk is $C^\phi_\eta(\alpha)=\eta e^{-\alpha}+(1-\eta)e^\alpha$. When $0<\eta<1$, its derivative vanishes when $\eta e^{-\alpha}=(1-\eta)e^\alpha$, giving $\alpha^*=\tfrac12\log(\eta/(1-\eta))$. At this score, the minimum is $H^\phi_\eta=2\sqrt{\eta(1-\eta)}$.

Suppose $\eta>1/2$. For every $\alpha\leq0$, the derivative $-\eta e^{-\alpha}+(1-\eta)e^\alpha$ is negative: $e^{-\alpha}\geq1$ and $e^\alpha\leq1$, so the positive term cannot outweigh $\eta e^{-\alpha}$. The wrong-or-tied risk is therefore minimized at zero, with value $1$. By symmetry, the same excess applies when $\eta<1/2$. Since $2\sqrt{\eta(1-\eta)}=\sqrt{1-m^2}$ for $m=|2\eta-1|$, the excess is $1-\sqrt{1-m^2}$. This increases with $m$, so

\delta_\phi(\varepsilon)=1-\sqrt{1-\varepsilon^2}.

For the scaled exponential loss $e^{-cu}$, the unrestricted minimizer is $\alpha^*/c$. Its calibration function remains the same, as the general theorem predicts.

Regularization Can Break the Equivalence

The unrestricted result relies on being able to replace every score $\alpha$ by $c\alpha$ or $\alpha/c$ without restriction or extra cost. In learning, scores often belong to a prescribed class and are penalized. For example, consider a population objective

J_c(f)=E[\phi(cYf(X))]+\lambda\Omega(f),

where $f$ belongs to a score class $\mathcal F$, $\lambda>0$, and $\Omega$ is a penalty. Setting $g=cf$ changes the objective to the original loss evaluated at $g$, but also changes the class and penalty. The next result states the exact relationship when those changes can be tracked.

Result: Under the substitution $g=cf$, the feasible scores become $c\mathcal F=\{cf:f\in\mathcal F\}$, and the penalty becomes $\Omega(g/c)$. If $c\mathcal F=\mathcal F$ and $\Omega(af)=a^q\Omega(f)$ for every $a>0$ and some $q>0$, then minimizing $J_c$ over $\mathcal F$ is equivalent, in the $g$ parameterization, to minimizing $$ E[\phi(Yg(X))]+\lambda c^{-q}\Omega(g) $$ over $\mathcal F$.

Derivation. For $g=cf$, one has $f=g/c$, and therefore

J_c(f)=E[\phi(Yg(X))]+\lambda\Omega(g/c) =E[\phi(Yg(X))]+\lambda c^{-q}\Omega(g).

The substitution is bijective between $\mathcal F$ and $c\mathcal F$; when the class is scale-invariant, these are the same feasible set. Thus the displayed objectives have corresponding minimizers. In particular, for a squared-norm penalty, $q=2$, scaling the loss by $c$ is equivalent to changing the penalty strength from $\lambda$ to $\lambda/c^2$, not to keeping it fixed. $\square$

Worked Example: Logistic Loss With Fixed Ridge Penalty

Let the score be a constant $f\in\mathbb R$, let $\eta=0.75$, and use logistic loss with the fixed penalty $f^2/2$. The objective for scale $c$ is $J_c(f)=0.75\log(1+e^{-cf})+0.25\log(1+e^{cf})+f^2/2$. Its derivative is

J_c'(f)=c\left(\frac{1}{1+e^{-cf}}-0.75\right)+f, \qquad J_c''(f)=c^2\sigma(cf)(1-\sigma(cf))+1>0.

The positive second derivative ensures a unique minimizer. For $c=1$, the derivative at $f=0.2$ is $\sigma(0.2)-0.75+0.2\approx-0.000166$, while at $f=0.201$ it is approximately $0.00108$. The minimizer is therefore between $0.2$ and $0.201$, approximately $0.200$.

For $c=2$, the derivative is $2(\sigma(2f)-0.75)+f$. At $f=0.25$ it is approximately $-0.00508$; at $f=0.26$ it is approximately $0.01429$. The unique minimizer is between $0.25$ and $0.26$, approximately $0.253$. Thus the fixed penalty changes the optimal score when the loss is rescaled. Under $g=cf$, the penalty would instead be $g^2/(2c^2)$; keeping $f^2/2$ fixed does not preserve the unrestricted rescaling equivalence.

Common Pitfall and Practical Interpretation

Wrong reasoning: “The calibration function is unchanged under positive scaling, so a fitted model and its guarantees are unchanged.” The first clause concerns unrestricted conditional excess risks. It does not imply that a constrained or penalized optimization problem has the same minimizer.

Correction: First identify the setting. With unrestricted real-valued scores, positive scaling changes the score minimizers by $1/c$, while preserving their signs, conditional minima, and calibration function. With a restricted class, check whether rescaling keeps scores feasible. With a penalty, transform the penalty too; for a degree-$q$ homogeneous penalty its effective strength changes by $c^{-q}$. If the class is not scale-invariant or the penalty is held fixed, the fitted scores may change.

A related distinction concerns derivatives. If $\phi$ is differentiable, then $\phi_c'(u)=c\phi'(cu)$; if it is twice differentiable, $\phi_c''(u)=c^2\phi''(cu)$. Such local derivative quantities can change with $c$, even though the unrestricted calibration function does not. They should not be substituted for the calibration function itself.

Key takeaway: Positive scaling is a change of score units for unrestricted conditional calibration. It becomes a change to the learning problem when a score class or a fixed regularizer prevents the corresponding rescaling.

Check Your Understanding

Use the definitions and distinctions in this tutorial to answer the following.

For $c>0$, how are the unrestricted conditional minimizer sets for $\phi(cu)$ and $\phi(u)$ related?
Why does the constraint defining wrong or tied scores remain unchanged under the substitution $t=c\alpha$?
For a squared-norm penalty, what happens to its effective coefficient after changing variables from $f$ to $g=cf$?
Does calibration-function invariance require $\phi$ to be convex or differentiable? Explain.
Give one reason a fixed regularizer can make the optimal score differ after loss scaling, even when the unrestricted conditional minimizer merely changes units.