Pointwise Calibration and Surrogate Consistency

From Conditional Guarantees to Population Regret

A calibration result is first a statement about one feature value at a time: if a score makes the wrong classification, its conditional surrogate risk cannot be arbitrarily close to optimal when the conditional classification regret is substantial. The population question is whether this local fact still controls average classification performance when the score and the conditional distribution vary across feature values.

The answer is yes under a precise condition. If a calibration function is continuous and strictly positive at every positive regret level, then a sequence of predictors whose population surrogate excess risk tends to zero must also have classification regret tending to zero. The proof uses a threshold argument: substantial regret can occur only on a set whose probability is controlled by the surrogate excess.

Conditional and Population Regret

Consider binary labels \(Y\in\{-1,+1\}\), a feature vector \(X\), and a measurable real-valued score function \(g\). Write \(\eta(x)=P(Y=+1\mid X=x)\), using a version of the conditional probability. Fix the decision rule \(h(s)=+1\) for \(s\geq 0\) and \(h(s)=-1\) for \(s<0\). The Bayes decision is \(+1\) when \(\eta(x)\geq 1/2\), with either choice optimal at equality.

In particular, \(0\leq r_\eta(s)\leq 1\). The population classification regret of \(g\) is \(R_0(h\circ g)-R_0^*=E[r_{\eta(X)}(g(X))]\), where \(R_0^*\) is the Bayes classification risk. This identity follows by conditioning the zero-one loss on \(X\) and subtracting the conditional Bayes risk.

For a margin loss \(\phi\), define the conditional surrogate risk and its infimum by \(C_\phi(\eta,s)=\eta\phi(s)+(1-\eta)\phi(-s)\) and \(C_\phi^*(\eta)=\inf_{u\in\mathbb R}C_\phi(\eta,u)\). The conditional surrogate excess at score \(s\) is \(e_\phi(\eta,s)=C_\phi(\eta,s)-C_\phi^*(\eta)\), which is nonnegative. Assume these quantities are measurable when evaluated at \(X\), and that the relevant expectations are finite.

Calibration Functions and the Pointwise Bound

A calibration function expresses how much conditional surrogate excess is unavoidable when the classification regret is at least a specified amount. For the fixed decision rule above, one standard definition is the lower envelope \(\psi(t)=\inf\{e_\phi(\eta,s):\eta\in[0,1],\,s\in\mathbb R,\,r_\eta(s)\geq t\}\) for \(0<t\leq1\), with \(\psi(0)=0\). An infimum over an empty set is \(+\infty\). Because the constraint sets shrink as \(t\) increases, \(\psi\) is nondecreasing.

The last inequality is the pointwise calibration bound. When the regret is positive, the pair \((\eta,s)\) belongs to the constraint set used to define \(\psi(r_\eta(s))\), so its excess cannot be smaller than that set’s infimum. When regret is zero, the inequality follows from nonnegative excess and \(\psi(0)=0\).

The Population Transfer Theorem

Let \(g_n\) be a sequence of measurable score functions. Assume the pointwise calibration bound holds almost surely for each \(g_n\), and assume the population surrogate excess is \(\delta_n=R_\phi(g_n)-R_\phi^*=E[e_\phi(\eta(X),g_n(X))]\). Here \(R_\phi(g)=E[\phi(Yg(X))]\), and the equality with the expected conditional excess is assumed to hold; it follows by conditioning on \(X\) when \(R_\phi^*=E[C_\phi^*(\eta(X))]\). This condition is met, for example, when the conditional infima can be approached by measurable score functions. The theorem concerns this population excess, not an empirical risk.

Proof. Write \(r_n=r_{\eta(X)}(g_n(X))\). The pointwise calibration inequality and the conditional-excess identity give \(E[\psi(r_n)]\leq E[e_\phi(\eta(X),g_n(X))]=\delta_n\). On the event \(\{r_n\geq a\}\), monotonicity gives \(\psi(r_n)\geq\psi(a)>0\). Thus \(\psi(a)\mathbf{1}\{r_n\geq a\}\leq\psi(r_n)\) everywhere, and taking expectations yields \(\psi(a)P(r_n\geq a)\leq\delta_n\). This proves the probability bound.

Since \(0\leq r_n\leq1\), split its expectation according to whether it is below \(a\). On \(\{r_n<a\}\), it is at most \(a\); on the complement, it is at most \(1\). Therefore \(E[r_n]\leq a+P(r_n\geq a)\leq a+\delta_n/\psi(a)\). For fixed \(a>0\), positivity makes the denominator nonzero, so taking the limit superior in \(n\) gives \(\limsup_n E[r_n]\leq a\). This holds for every \(a\in(0,1]\); letting \(a\) decrease to zero gives \(E[r_n]\to0\). Since \(E[r_n]=R_0(h\circ g_n)-R_0^*\), the claimed conclusion follows. \(\square\)

Continuity is part of the usual regularity condition on a calibration function and ensures it has no jump at zero. The threshold proof itself uses positivity at each fixed positive threshold, along with monotonicity; it does not require differentiability or convexity. In particular, it avoids applying Jensen’s inequality to a function that may not be convex.

Worked Population Consequences

What the Implication Does—and Does Not—Say

The theorem separates two claims. Calibration supplies a pointwise link between conditional surrogate excess and conditional classification regret. Averaging that link, then using positivity away from zero, transfers a vanishing population surrogate excess into vanishing population classification regret. Continuity and positivity are properties of the calibration function, not assumptions that the feature distribution has a density or that the conditional probability stays away from \(1/2\).

The premise is important: the theorem starts with a sequence whose population surrogate excess tends to zero. It does not by itself establish that a particular fitting procedure produces such a sequence. Nor does it say that a small finite-sample empirical surrogate risk is automatically a small population excess. Those are separate questions from the population transfer proved here.

• Pitfall: applying Jensen’s inequality without checking curvature. From \(E[\psi(r)]\leq\delta\), it is not generally valid to conclude \(\psi(E[r])\leq\delta\); that direction would require suitable convexity. Use the threshold event and the bound \(P(r\geq a)\leq\delta/\psi(a)\) instead.
• Pitfall: changing the decision rule while keeping the same calibration function. The logistic formula above was derived for the specified sign rule. A different map from scores to labels can alter which scores count as errors and therefore alter the constrained infimum.
• Pitfall: treating a rounded number as an exact risk. Keep the exact conditional excess in finite-sample or finite-\(n\) equalities, and mark decimal evaluations as approximations.