Why Calibration Alone Does Not Ensure Learning Consistency

Calibration Is a Property of a Loss, Not a Guarantee About an Algorithm

A calibrated surrogate loss gives a precise way to transfer small surrogate excess risk into small classification regret. The qualification “small surrogate excess risk” matters: calibration does not say that a learning procedure will achieve it. The procedure may be restricted to an inadequate score class, may fail to optimize its empirical objective, or may fit the observed sample without generalizing.

This distinction separates two questions. Calibration asks what follows if surrogate regret is small. Learning consistency asks whether the learner’s regret actually becomes small as the sample grows. To connect the questions, we need to track the errors incurred between the population optimum and the score returned by the learning procedure.

Definition: Let \(X\) be a feature variable, \(Y\in\{-1,+1\}\) a label, and \(\phi:\mathbb{R}\to[0,\infty)\) a margin loss. For a measurable score function \(f\), define its population surrogate risk by \(R_\phi(f)=E[\phi(Yf(X))]\). Define the unrestricted surrogate risk \(R_\phi^*=\inf_f R_\phi(f)\), where the infimum is over measurable score functions, and the classification risk \(R_{01}(g)=P(g(X)\ne Y)\). The Bayes classification risk is \(R_{01}^*=\inf_g R_{01}(g)\). For a score \(f\), use the classifier \(g_f(x)=+1\) if \(f(x)\ge 0\), and \(g_f(x)=-1\) otherwise.

The surrogate excess risk is \(R_\phi(f)-R_\phi^*\), while the classification excess risk, or classification regret, is \(R_{01}(g_f)-R_{01}^*\). For a calibrated loss, the population calibration result supplies a function \(\psi\), with \(\psi(0)=0\) and \(\psi(t)>0\) for \(t>0\), such that

\psi\!\left(R_{01}(g_f)-R_{01}^*\right) \le R_\phi(f)-R_\phi^*.

This inequality is the relevant transfer statement: if the right side tends to zero, the classification regret must tend to zero. It does not assert that the right side tends to zero for any particular data-dependent choice of \(f\).

Three Sources of Surrogate Excess Risk

A learner usually searches within a score class \(\mathcal{F}\), rather than among all measurable functions. It also uses a finite sample and may not minimize the empirical objective exactly. Write the observed independent and identically distributed sample as \((X_i,Y_i)_{i=1}^n\), and define

\widehat R_{\phi,n}(f)=\frac{1}{n}\sum_{i=1}^n \phi(Y_i f(X_i)).

For a learner returning \(\hat f_n\in\mathcal{F}\), the three error sources are:

Approximation error: \(R_\phi(\mathcal{F})-R_\phi^*\), where \(R_\phi(\mathcal{F})=\inf_{f\in\mathcal{F}}R_\phi(f)\). This measures what is lost by restricting the score class.
Estimation error: the discrepancy between empirical and population risks over \(\mathcal{F}\). A useful control is \(\sup_{f\in\mathcal{F}}|\widehat R_{\phi,n}(f)-R_\phi(f)|\).
Optimization error: the learner’s empirical objective above its best achievable value in \(\mathcal{F}\), namely \(\widehat R_{\phi,n}(\hat f_n)-\inf_{f\in\mathcal{F}}\widehat R_{\phi,n}(f)\).

The approximation term is a property of the class and the data distribution; the other two terms can depend on the sample and the algorithm. For each fixed \(n\), the following bound makes their roles explicit.

Theorem: Suppose \(\hat f_n\in\mathcal{F}\), the population and empirical risks are finite, and the quantities below are measurable (or are interpreted using measurable upper bounds). Then

R_\phi(\hat f_n)-R_\phi^* \le \underbrace{R_\phi(\mathcal{F})-R_\phi^*}_{\text{approximation}} + \underbrace{\widehat R_{\phi,n}(\hat f_n)-\inf_{f\in\mathcal{F}}\widehat R_{\phi,n}(f)}_{\text{optimization}} + 2\underbrace{\sup_{f\in\mathcal{F}}\left|\widehat R_{\phi,n}(f)-R_\phi(f)\right|}_{\text{estimation control}}.

Proof. Set \(D_n=\sup_{f\in\mathcal{F}}|\widehat R_{\phi,n}(f)-R_\phi(f)|\). For every \(f\in\mathcal{F}\), the definition of \(D_n\) gives \(R_\phi(f)\le \widehat R_{\phi,n}(f)+D_n\). Taking the infimum over \(f\) yields

R_\phi(\mathcal{F})\le \inf_{f\in\mathcal{F}}\widehat R_{\phi,n}(f)+D_n.

Also \(R_\phi(\hat f_n)\le\widehat R_{\phi,n}(\hat f_n)+D_n\). Subtract \(R_\phi(\mathcal{F})\) from this inequality and use the preceding lower bound on \(R_\phi(\mathcal{F})\). The result is

R_\phi(\hat f_n)-R_\phi(\mathcal{F}) \le \widehat R_{\phi,n}(\hat f_n)-\inf_{f\in\mathcal{F}}\widehat R_{\phi,n}(f)+2D_n.

Finally, add \(R_\phi(\mathcal{F})-R_\phi^*\) to both sides. This gives the stated inequality. \(\square\)

The bound is useful even when its terms do not vanish: it identifies why calibration alone is insufficient. A vanishing optimization gap cannot compensate for a class whose approximation error stays positive; a rich class cannot compensate for poor optimization; and exact empirical minimization does not by itself control estimation error.

When the Error Terms Vanish

The calibration transfer yields a direct consistency conclusion. Here consistency means that the learner’s classification regret converges to zero in probability. The loss must be calibrated for the score-to-classifier convention being used, and the relevant population calibration inequality must hold for the distribution under study.

Corollary: Suppose the calibration inequality above holds for every score output by the learner. If \(R_\phi(\hat f_n)-R_\phi^*\xrightarrow{p}0\), then \(R_{01}(g_{\hat f_n})-R_{01}^*\xrightarrow{p}0\). In particular, this conclusion follows if the approximation error tends to zero and the optimization and estimation-control terms in the theorem tend to zero in probability.

Proof. Fix \(\epsilon>0\). Since \(\psi(\epsilon)>0\) and \(\psi\) is nondecreasing, the calibration inequality implies that

\{R_{01}(g_{\hat f_n})-R_{01}^*\ge\epsilon\} \subseteq \{R_\phi(\hat f_n)-R_\phi^*\ge\psi(\epsilon)\}.

The probability of the event on the right tends to zero by the assumed convergence in probability of the surrogate excess risk. This holds for every \(\epsilon>0\), proving convergence in probability of the classification regret. The theorem’s bound shows that the stated vanishing error terms imply the required convergence of surrogate excess risk. \(\square\)

If the errors vanish almost surely instead, the same pointwise inequality gives almost-sure classification consistency. More generally, the mode of convergence in the conclusion follows from the mode in which the surrogate excess risk vanishes. Calibration supplies the transfer; the error analysis supplies the premise.

Worked Examples: How Consistency Can Fail

Worked Example: A Calibrated Loss With Persistent Approximation Error

Let \(X\) be equally likely to be \(-1\) or \(+1\), and let \(Y=X\) deterministically. Use the logistic margin loss \(\phi(u)=\log(1+e^{-u})\), and restrict the learner to constant scores \(f_c(x)=c\), \(c\in\mathbb{R}\).

The population surrogate risk of \(f_c\) is

R_\phi(f_c)=\frac12\log(1+e^{-c})+\frac12\log(1+e^c).

This function is minimized at \(c=0\): its derivative is \(\frac12\tanh(c/2)\), which is negative for \(c<0\), zero at \(0\), and positive for \(c>0\). Its minimum is

R_\phi(\mathcal{F})=R_\phi(f_0)=\log 2.

The unrestricted infimum is \(R_\phi^*=0\), since the score \(f(x)=a x\) has risk \(\log(1+e^{-a})\), which tends to zero as \(a\) tends to infinity. Thus the approximation error of the constant-score class is \(\log 2\). The Bayes classifier has zero classification risk. But \(f_0\) predicts \(+1\) at both feature values under the stated tie convention, so its classification risk is \(1/2\). The class cannot express the feature-dependent decision rule, no matter how accurately its empirical objective is optimized.

Worked Example: A Learner That Does Not Optimize

Let \(X\) be constant and \(Y=+1\) with probability one. Use logistic loss and let \(\mathcal{F}\) contain every constant score \(f_c(x)=c\), \(c\in\mathbb{R}\). The unrestricted risk and the class infimum are both zero, because

R_\phi(f_c)=\log(1+e^{-c})\longrightarrow 0 \quad\text{as }c\longrightarrow+\infty.

For every sample, the empirical infimum is also zero. Suppose an algorithm nevertheless returns \(f_{-1}\) for every sample size. Its empirical optimization gap is

\widehat R_{\phi,n}(f_{-1})-\inf_{f\in\mathcal{F}}\widehat R_{\phi,n}(f) =\log(1+e)-0 =\log(1+e).

This gap does not shrink. The output predicts \(-1\), so its classification risk is \(1\), while the Bayes risk is \(0\). The approximation error is zero, but persistent optimization error prevents the learner from approaching the surrogate infimum. The example also illustrates that an infimum need not be attained: that fact does not prevent us from defining the optimization gap relative to the infimum.

Worked Example: Empirical Fit Without Generalization

Let \(X\) be uniform on \([0,1]\) and \(Y=+1\) with probability one. Use logistic loss and consider a class containing all positive integer constant scores and, for every finite set \(A\subset[0,1]\) and positive integer \(k\), the score

f_{A,k}(x)= \begin{cases} k, & x\in A,\\ -1, & x\notin A. \end{cases}

These functions are measurable because finite sets are measurable. The class has zero approximation error: its constant scores \(k\) have risks \(\log(1+e^{-k})\), which tend to zero. Given a sample of size \(n\), let \(A_n=\{X_1,\ldots,X_n\}\) and return \(\hat f_n=f_{A_n,n}\). At every observed feature value the score is \(n\), so

\widehat R_{\phi,n}(\hat f_n)=\log(1+e^{-n})\longrightarrow 0.

The empirical infimum is zero, since the class permits arbitrarily large positive integer scores on the finite set of observed points. Thus the learner’s empirical optimization gap is \(\log(1+e^{-n})\), which tends to zero. But \(A_n\) is finite, so \(P(X\in A_n)=0\) for each realized sample. Almost surely, a new \(X\) is outside \(A_n\), where the score is \(-1\). Consequently, the population surrogate risk is \(\log(1+e)\), and the classification risk is \(1\), for every \(n\). The failure is uncontrolled estimation: excellent empirical performance does not imply small population risk for this class.

Common Pitfall and Practical Interpretation

A common mistake is to reason, “The loss is calibrated, so empirical risk minimization is classification-consistent.” This skips the essential premise. Calibration transfers population surrogate excess risk; empirical minimization is about a sample objective, and the fitted class may also be inadequate. The corrected argument checks each link: the class can approximate the unrestricted surrogate infimum, empirical risk is sufficiently close to population risk over the class, and the algorithm’s empirical optimization gap is small.

The three examples isolate different failures. In the first, the class restriction leaves a positive approximation error even if optimization and estimation are ideal. In the second, the class is rich enough but the algorithm keeps a positive optimization gap. In the third, the class can approximate the target and the learner nearly minimizes empirical risk, yet estimation error remains uncontrolled. Calibration is necessary for this particular surrogate-to-classification transfer, but it does not remove any of these learning errors.

Key takeaway: Calibration is a property of the loss that controls classification regret by surrogate excess risk. Learning consistency requires a separate argument that the learner’s population surrogate excess risk vanishes; approximation, estimation, and optimization errors determine whether it does.

Check Your Understanding

Use the definitions and error decomposition to answer the following.

What does calibration imply when a sequence of scores has surrogate excess risk tending to zero, and what does it not imply about an arbitrary learning algorithm?
In the constant-score example, calculate the approximation error and explain why improving empirical optimization cannot remove it.
For the learner that always returns score \(-1\) when \(Y=+1\), why is its optimization gap positive even though the class has zero approximation error?
In the finite-set example, which error source fails to vanish, and why does the empirical risk still tend to zero?
State sufficient conditions, in terms of the three error contributions, for a calibrated surrogate learner to be classification-consistent in probability.