Excess-Risk Transfer Through Calibration Functions

From Conditional Comparison to Risk Transfer

A surrogate loss is useful for classification when its excess risk controls the cost of making classification errors. The first comparison is local: at a fixed feature value, how much surrogate risk is paid when the score induces a non-Bayes decision? A calibration function expresses this comparison in a scale that can be averaged over the feature distribution.

The central step is not to average the conditional inequality mechanically. Convexity matters: Jensen’s inequality lets the average conditional surrogate excess control the calibration function evaluated at the average classification regret. This gives a population guarantee without requiring every feature value to have the same conditional difficulty.

Conditional Regret and Surrogate Excess

Let $Y\in\{-1,+1\}$, let $X$ be the feature vector, and define $\eta(x)=P(Y=+1\mid X=x)$. A real-valued measurable score $f$ predicts $+1$ when $f(x)\geq 0$ and predicts $-1$ otherwise. Write $v=f(x)$ and $\eta=\eta(x)$ when considering one fixed feature value.

The conditional zero-one risk of a decision is the probability of a classification error given $X=x$. Its minimum is $\min\{\eta,1-\eta\}$, attained by predicting the more probable class. With the stated tie convention, define $D_\eta(v)=1$ when the score’s decision is not a conditional Bayes decision, and $D_\eta(v)=0$ otherwise. In particular, when $\eta=1/2$, either decision is Bayes and $D_\eta(v)=0$.

Definition: The conditional classification regret of score $v$ is $$ r_\eta(v)=|2\eta-1|D_\eta(v). $$ For a margin loss $\phi$, the conditional surrogate risk and its conditional infimum are $$ C_\eta(v)=\eta\phi(v)+(1-\eta)\phi(-v), \qquad H_\eta^*=\inf_{a\in\mathbb R} C_\eta(a). $$ The conditional surrogate excess is $\Delta_\eta(v)=C_\eta(v)-H_\eta^*$, whenever $C_\eta(v)$ is finite.

The regret formula follows directly from the two possible conditional decisions. If $\eta>1/2$, predicting $-1$ has conditional error probability $\eta$, whereas predicting $+1$ has error probability $1-\eta$; their difference is $2\eta-1$. If $\eta<1/2$, the corresponding difference is $1-2\eta$. At $\eta=1/2$ the difference is zero. Thus a wrong decision costs $|2\eta-1|$, and a correct Bayes decision costs zero regret.

The Calibration-Function Inequality

The constrained-infimum viewpoint compares the surrogate risk of scores that incur at least a specified amount of classification regret with the unrestricted conditional surrogate infimum. This comparison gives a sharp lower envelope. A useful calibration function may be that envelope itself when it is convex, or a convex lower bound on it.

Definition: The constrained-infimum envelope is $$ \Psi(t)=\inf\{\Delta_\eta(v):\eta\in[0,1],\ v\in\mathbb R,\ r_\eta(v)\geq t\}, \qquad 0\leq t\leq 1, $$ where the infimum of an empty set is $+\infty$. A convex calibration function $\psi$ is a finite, nonnegative, convex, nondecreasing function on $[0,1]$ such that $\psi(t)\leq\Psi(t)$ for every $t$. In particular, $\psi(0)=0$ is the usual normalization.

Theorem: For every $\eta\in[0,1]$ and score $v$ with finite conditional surrogate risk, $$ \Delta_\eta(v)\geq \Psi(r_\eta(v))\geq\psi(r_\eta(v)). $$

Proof. Fix $\eta$ and $v$, and put $t=r_\eta(v)$. The pair $(\eta,v)$ belongs to the set used to define $\Psi(t)$, because its regret is at least $t$ (in fact, equal to $t$). An infimum over that set cannot exceed the value $\Delta_\eta(v)$ at this particular pair. Therefore $\Delta_\eta(v)\geq\Psi(t)$. The defining minorant property gives $\Psi(t)\geq\psi(t)$. Substituting $t=r_\eta(v)$ proves both inequalities. $\square$

This argument is the pointwise transfer: a score with substantial conditional classification regret must pay at least the corresponding calibration-function amount in conditional surrogate excess. The envelope is nondecreasing because increasing $t$ shrinks the set over which the infimum is taken. Convexity, however, does not follow merely from this nesting; it is an additional property of the calibration function used for population transfer.

Worked Example: Checking a Conditional Bound

Suppose a fitted score is evaluated at a feature value where $\eta=0.8$, and its score is negative. The conditional Bayes decision is $+1$, so the fitted decision is wrong. Suppose the surrogate analysis supplies the calibration function $\psi(t)=t^2/2$, and the conditional surrogate excess at this feature value is $0.22$.

The conditional classification regret is $$ r_{0.8}(v)=|2(0.8)-1|=|1.6-1|=0.6. $$ The calibration requirement at this regret is $$ \psi(0.6)=\frac{(0.6)^2}{2}=\frac{0.36}{2}=0.18. $$ The reported surrogate excess satisfies the pointwise inequality because $0.22\geq0.18$. This check is conditional: it compares the quantities at the same feature value, before taking any expectation over $X$.

Convexity Gives the Population Bound

Let $f(X)$ be a measurable score and write $r(X)=r_{\eta(X)}(f(X))$ and $\Delta(X)=\Delta_{\eta(X)}(f(X))$. Define the population classification regret as $\mathcal E_{01}(f)=\mathbb E[r(X)]$, and the population conditional surrogate excess as $\mathcal E_\phi(f)=\mathbb E[\Delta(X)]$. When the infimum of the population surrogate risk equals $\mathbb E[H_{\eta(X)}^*]$, as under the usual measurable-minimization conditions for the score class, $\mathcal E_\phi(f)$ is the usual population excess surrogate risk.

Theorem: Suppose the conditional calibration inequality $\Delta_{\eta(x)}(f(x))\geq\psi(r_{\eta(x)}(f(x)))$ holds almost surely for a convex calibration function $\psi:[0,1]\to[0,\infty)$. Assume $r(X)$ and $\Delta(X)$ are measurable, $\mathbb E[\Delta(X)]<\infty$, and $\mathbb E[H_{\eta(X)}^*]<\infty$. Then $$ \psi\bigl(\mathcal E_{01}(f)\bigr)\leq\mathcal E_\phi(f). $$

Proof. The pointwise calibration inequality holds almost surely, so monotonicity of expectation gives $$ \mathbb E[\Delta(X)]\geq\mathbb E[\psi(r(X))]. $$ Because $r(X)\in[0,1]$, it is integrable. The function $\psi$ is convex, and $\psi(r(X))$ is measurable; Jensen’s inequality therefore gives $$ \mathbb E[\psi(r(X))]\geq\psi(\mathbb E[r(X)]). $$ Combining the two inequalities and using the definitions of the population excesses yields $$ \mathcal E_\phi(f)\geq\psi(\mathcal E_{01}(f)). $$ This proves the stated bound. $\square$

The direction is important: the surrogate excess is bounded below by the calibration function of the classification regret. The theorem does not say that the surrogate excess is at most that function. Nor does convexity itself establish the conditional calibration inequality; it is used after that inequality has been obtained, to compare the mean of $\psi(r(X))$ with $\psi$ of the mean regret.

Worked Example: Two Feature Strata

Suppose $X$ falls in stratum A with probability $1/4$ and stratum B with probability $3/4$. Their conditional classification regrets are $0.2$ and $0.6$, respectively. Suppose $\psi(t)=t^2/2$, and their conditional surrogate excesses are $0.04$ and $0.20$. Check the conditional inequalities and the population conclusion.

In stratum A, $\psi(0.2)=0.2^2/2=0.02$, and $0.04\geq0.02$. In stratum B, $\psi(0.6)=0.6^2/2=0.18$, and $0.20\geq0.18$. Thus the pointwise requirements hold in both strata.

The population classification regret and surrogate excess are $$ \mathcal E_{01}(f)=\frac14(0.2)+\frac34(0.6)=0.05+0.45=0.50, $$ $$ \mathcal E_\phi(f)=\frac14(0.04)+\frac34(0.20)=0.01+0.15=0.16. $$ The population bound is $\psi(0.50)=0.50^2/2=0.125\leq0.16$. Jensen’s intermediate comparison is also visible: $$ \mathbb E[\psi(r(X))]=\frac14(0.02)+\frac34(0.18)=0.005+0.135=0.14 \geq0.125=\psi(\mathbb E[r(X)]). $$

Worked Example: Continuously Varying Conditional Regret

Let $X$ be uniform on $[0,1]$. Suppose the conditional regret is $r(X)=X$, the calibration function is $\psi(t)=t^2/2$, and the conditional surrogate excess is $\Delta(X)=X^2/2$. The pointwise calibration inequality holds with equality at every $x$.

The mean classification regret is $$ \mathcal E_{01}(f)=\int_0^1 x\,dx=\left[\frac{x^2}{2}\right]_0^1=\frac12. $$ The mean surrogate excess is $$ \mathcal E_\phi(f)=\int_0^1\frac{x^2}{2}\,dx =\left[\frac{x^3}{6}\right]_0^1=\frac16. $$ Applying the population calibration function to the mean regret gives $$ \psi(\mathcal E_{01}(f))=\frac{(1/2)^2}{2}=\frac18. $$ Thus $1/8\leq1/6$, as required. The Jensen step is strict here: $\mathbb E[r(X)^2]/2=1/6$, while $(\mathbb E[r(X)])^2/2=1/8$, because the conditional regret varies across feature values.

Common Pitfall and Interpretation

A common mistake is to average the conditional regrets and then average the calibration function without checking the direction of Jensen’s inequality. For convex $\psi$, the correct relation is $\psi(\mathbb E[r])\leq\mathbb E[\psi(r)]$, not the reverse. Combined with the pointwise bound, this gives $\psi(\mathbb E[r])\leq\mathbb E[\Delta]$. Reversing Jensen’s inequality would incorrectly claim that the calibration function of the mean regret is at least the average conditional calibration value.

Another mistake is to treat a conditional guarantee as if it already were a population guarantee. The conditional inequality must hold at almost every feature value under the data distribution, and the quantities must be measurable and integrable for the expectations to be used. Convexity supplies the averaging step; it does not repair a missing pointwise bound or undefined expectation.

Key takeaway: A pointwise calibration inequality says $\Delta(X)\geq\psi(r(X))$. If $\psi$ is convex, taking expectations and applying Jensen yields $\psi(\mathbb E[r(X)])\leq\mathbb E[\Delta(X)]$. The conditional comparison and the convex averaging step are distinct parts of the argument.

Check Your Understanding

Use the definitions and inequalities above to answer these questions.

For $\eta=0.7$, what is the conditional regret of a score that predicts the wrong class?
Why does the constrained-infimum envelope $\Psi(t)$ increase (or stay constant) as $t$ increases?
If $\Delta(X)\geq\psi(r(X))$ almost surely, which direction does Jensen’s inequality give when $\psi$ is convex?
In the two-stratum example, calculate $\mathbb E[\psi(r(X))]$ and compare it with $\psi(\mathbb E[r(X)])$.
What additional role does measurability play in the population transfer proof?