Cross-Fitted Calibration of Discriminant Scores

Why Calibrate Discriminant Scores?

A discriminant rule may rank cases well yet produce probabilities that are too extreme, too conservative, or otherwise poorly matched to observed frequencies. Calibration estimates a map from a classifier's score to a probability. The map is fitted using labeled observations, so a central question is whether each score was produced without training the classifier on that same observation. If not, the apparent score-label relationship can be overly favorable.

Cross-fitting addresses this source of optimism: each observation receives a score from a model fitted without that observation. A separate, untouched evaluation set is still needed to assess the calibrated procedure. Cross-fitting prevents direct reuse of an observation in its score-generating model; it does not make the calibration fit independent of the labels used to fit it, nor does it guarantee good probability estimates.

Definition: For binary class label $Y\in\{0,1\}$, a score $S$ is a numerical output from a discriminant rule, oriented so larger values indicate stronger evidence for class 1. A calibration map $g$ transforms a score into a predicted probability $g(S)\in[0,1]$. A population calibration property is $E[Y\mid g(S)]=g(S)$; fitting a map on a finite sample does not by itself establish this property in the population.

Out-of-Fold Scores and the Data Roles

Suppose the development data are $(X_i,Y_i)$, $i=1,\ldots,n$, and are divided into $K$ folds $I_1,\ldots,I_K$. For each fold $k$, fit the base discriminant model using observations outside $I_k$. Score each observation $i\in I_k$ with that fitted model. Denote the resulting score $S_i^{\mathrm{OOF}}$. Thus the model producing $S_i^{\mathrm{OOF}}$ did not use $(X_i,Y_i)$ in its fit.

Fit the calibration map using the pairs $(S_i^{\mathrm{OOF}},Y_i)$. The calibration sample is not an evaluation sample: all its labels participate in estimating $g$. To estimate the performance of the final calibrated procedure, keep an independent evaluation set out of base-model fitting, calibration fitting, and choices such as selecting between Platt and isotonic calibration. On that set, apply the complete fitted procedure and calculate the chosen performance measure.

Set aside evaluation data.
Do this before choosing or fitting the calibration method. Do not use these observations to fit the base model or the calibration map.

Generate out-of-fold development scores.
For each fold, train the base model on the other folds and score only the held-out fold.

Fit the calibration map.
Use the out-of-fold scores and their observed labels, applying the same score orientation and preprocessing intended for use.

Evaluate the fixed procedure.
Apply the base-model and calibration steps to the untouched data, then report the evaluation measure without using the result to retune the procedure.

A common implementation fits the base model on all development observations after generating out-of-fold scores, then applies the learned calibration map to that full-data model's scores. This is convenient, but the full-data model may produce scores on a different scale or with a different distribution from models trained on only $K-1$ folds. The independent evaluation set assesses the actual combination being used, including any such mismatch; it does not make the mismatch disappear. One alternative is to deploy a procedure whose score-generating models match the cross-fitting setup, for example a pre-specified fold-based ensemble. The deployed procedure and evaluation procedure should agree.

The usual independence argument presumes independent observations and folds chosen without looking at outcomes in a way that leaks information. For clustered or longitudinal data, split at the independent-unit level; random folds of correlated records can put near-duplicates on both sides. Stratification can help ensure class representation in each fold, but it does not replace a genuinely untouched evaluation set.

Two Calibration Maps

Definition: Platt scaling fits a logistic regression of the binary label on the score: $$ g(s)=\frac{1}{1+\exp\{-(a+bs)\}}. $$ Here $a$ is an intercept and $b$ is a slope. For a score whose larger values indicate class 1, a positive $b$ preserves that orientation. If an input score is already a probability, it is common to use its logit as the predictor; scores equal to 0 or 1 require clipping or a different finite score representation.

Platt scaling imposes a smooth, two-parameter sigmoid relationship. It can be relatively stable when the calibration sample is modest, but that stability comes from the assumed form: a truly irregular relationship may not be represented well. With both classes present and finite maximum-likelihood estimates, the intercept and slope can be found by maximizing the Bernoulli log-likelihood. Complete or quasi-complete separation can prevent finite maximum-likelihood estimates.

Definition: Isotonic calibration estimates a nondecreasing function $g$ by minimizing the calibration-sample squared error over nondecreasing fitted values: $$ \min_{g:\,s_i\le s_j\Rightarrow g(s_i)\le g(s_j)} \sum_{i=1}^{n}(Y_i-g(S_i^{\mathrm{OOF}}))^2, \qquad 0\le g(S_i^{\mathrm{OOF}})\le 1. $$ At tied scores, the fitted value is shared. The pool-adjacent-violators algorithm combines neighboring score blocks whenever their fitted averages violate the required order.

Isotonic calibration does not impose a sigmoid shape, but it can create a step function with sharp changes. In a small calibration sample, a few labels can determine large portions of the fitted map, including fitted probabilities of zero or one. The greater flexibility is not automatically an advantage.

Worked Example: Platt Scaling From a Balanced Score Sample

Suppose eight out-of-fold scores are $-1$ and eight are $+1$. Among the cases with score $-1$, two have $Y=1$; among those with score $+1$, six have $Y=1$. Fit the Platt map and verify its likelihood equations.

For a logistic model with an intercept, the log-likelihood is $$ \ell(a,b)=\sum_{i=1}^{16}\left[Y_i(a+bS_i)-\log\{1+\exp(a+bS_i)\}\right]. $$ The observed positive fraction is $2/8=1/4$ at score $-1$ and $6/8=3/4$ at score $+1$. A candidate fit is $a=0$ and $b=\log 3$, since $$ g(-1)=\frac{1}{1+\exp(\log 3)}=\frac14, \qquad g(+1)=\frac{1}{1+\exp(-\log 3)}=\frac34. $$

To check that this is the maximum-likelihood fit, the derivative with respect to $a$ is $\sum_i(Y_i-g(S_i))$. The residual sum at score $-1$ is $2-8(1/4)=0$, and at score $+1$ it is $6-8(3/4)=0$, so the intercept score is zero. The derivative with respect to $b$ is $\sum_i S_i(Y_i-g(S_i))$, also zero because each score group has zero residual sum. The logistic log-likelihood is concave in $(a,b)$, so a finite stationary point is a global maximizer. The fitted probabilities reproduce the two observed group proportions, but this small-sample agreement is not proof of population calibration.

What the Fitting Equations Do—and Do Not—Guarantee

When Platt scaling includes an intercept and has a finite maximum-likelihood fit, its intercept score equation gives a useful exact sample identity. Write $p_i=g(S_i^{\mathrm{OOF}})$ and let $\bar Y=n^{-1}\sum_iY_i$. Differentiating the log-likelihood with respect to $a$ gives $$ \frac{\partial\ell}{\partial a}=\sum_{i=1}^{n}(Y_i-p_i). $$ At an interior maximum, this derivative is zero. Therefore, $$ \frac{1}{n}\sum_{i=1}^{n}p_i=\frac{1}{n}\sum_{i=1}^{n}Y_i=\bar Y. $$ This is agreement of the average fitted probability with the sample prevalence in the calibration data. The slope equation also gives $\sum_i S_i^{\mathrm{OOF}}(Y_i-p_i)=0$, but neither equation says that $E[Y\mid g(S)]=g(S)$ in new data. In particular, average agreement is weaker than calibration across the range of predictions.

Result: Suppose the evaluation observations $(X_i^{\mathrm{test}},Y_i^{\mathrm{test}})$, $i=1,\ldots,m$, are independent and identically distributed from a target population and are independent of all data used to fit the deployed prediction procedure. For a fixed fitted procedure $\hat q$, define the evaluation loss $L_i=L(Y_i^{\mathrm{test}},\hat q(X_i^{\mathrm{test}}))$, where the loss has finite expectation. Then $m^{-1}\sum_iL_i$ is conditionally unbiased for the procedure's population risk, given the development data.

Proof. Condition on all development data and hence on the fitted procedure $\hat q$. Independence means each evaluation observation still has the target-population distribution under this conditioning. Thus $$ E\left[\left.\frac1m\sum_{i=1}^{m}L_i\right|\text{development data}\right] =\frac1m\sum_{i=1}^{m}E[L_i\mid\text{development data}] =E_{(X,Y)}[L(Y,\hat q(X))\mid\text{development data}]. $$ The right-hand side is the conditional population risk of the fitted procedure. Finite expectation justifies the expectations in the sum. This establishes unbiasedness for that procedure, not for a different model or a procedure selected after inspecting evaluation results. $\square$

The same argument applies to bounded losses such as misclassification loss and Brier loss. For log loss, the finite-expectation condition matters: an unprotected prediction of zero for an observed positive label, or one for an observed negative label, gives infinite loss. Most importantly, the result concerns an independent evaluation sample. Out-of-fold scores alone do not make the calibration sample an unbiased test set, because its labels were used to estimate the map.

Worked Example: Constructing Out-of-Fold Scores

Consider six development observations divided into three folds, each containing two observations. For fold 1, train on observations 3–6 and score observations 1–2; for fold 2, train on 1–2 and 5–6 and score 3–4; for fold 3, train on 1–4 and score 5–6. Suppose the resulting score-label pairs are:

Observation	Fold	Out-of-fold score	Label $Y$
1	1	-1.2	0
2	1	0.4	1
3	2	-0.5	0
4	2	1.1	1
5	3	-0.2	1
6	3	0.8	0

Each score is produced by a model that excludes the observation it scores. The score for observation 5, for instance, comes from the model trained on observations 1–4, not a model fitted using its label $Y_5=1$. All six pairs may now be used to fit a calibration map. They may not also be described as an independent test set: the labels in the last column directly determine the fitted map.

Suppose a separate test set has 100 independent observations and the fitted calibrated procedure misclassifies 18. Its test error is $18/100=0.18$. Under the stated sampling assumptions, this estimates the risk of the deployed procedure. If the analyst uses these 100 labels to choose between Platt and isotonic calibration and then reports the lower error as though the choice had been pre-specified, the reported value no longer has the simple unbiasedness interpretation: selection used the evaluation outcomes.

Worked Example: Isotonic Pooling and Small-Sample Extremes

Suppose five calibration observations, ordered by increasing out-of-fold score, have labels $0,1,0,1,1$. Isotonic fitted values must be nondecreasing. The initial label sequence decreases from 1 to 0 between the second and third observations, so those adjacent observations must be pooled. Their average is $(1+0)/2=1/2$. The resulting fitted sequence is $$ 0,\quad \frac12,\quad \frac12,\quad 1,\quad 1. $$

This sequence is nondecreasing and minimizes the squared error subject to that constraint. In particular, the first singleton block has mean 0, the pooled middle block has mean $1/2$, and the final two labels form a block with mean 1. The block means are ordered, so no further pooling is required by the pool-adjacent-violators procedure. The fitted map assigns exact probabilities 0 and 1 to parts of the score range on the evidence of only one or two observations. Such extremes can yield severe log loss if future outcomes contradict them. A larger calibration sample, a smoother method such as Platt scaling, or a pre-specified probability bound may be preferable; any method choice should be evaluated without reusing the evaluation labels.

Diagnosing Overfitting and Preserving Evaluation

Calibration methods can overfit in different ways. Isotonic calibration is especially vulnerable when few observations occupy each score region: many small blocks create a jagged step function, and the end blocks can be based on very few cases. Platt scaling uses fewer parameters, but it can still be unstable when the calibration sample has few events, a narrow score range, or separation. A fitted slope with the wrong sign can also indicate that the score orientation is reversed, or that sampling variation is substantial.

Inspect the number of observations and both class counts in the calibration sample, and how many cases support each isotonic block.
Compare fitted probabilities with observed event fractions in broad, pre-specified score ranges; avoid interpreting tiny bins as reliable evidence.
Check whether the calibration sample's score range and scale resemble those of the model that will be deployed.
Keep the evaluation sample independent, and do not repeatedly adjust the calibration method in response to its results.

Cross-fitting is a way to generate less optimistically biased training scores for the calibration step, not a cure for small samples, model misspecification, dependence, or distribution change. An honest assessment concerns the complete procedure: score production, calibration, and any pre-specified decisions about how that calibrated probability is evaluated. Classification thresholds and the consequences of different classification errors are separate decision questions; probability calibration should not be confused with choosing such a threshold.

Common pitfall: Fitting a discriminant model on all observations, using its in-sample scores to fit a calibrator, and then treating the calibration fit as if it were based on new predictions. The scores have been influenced by the same labels used to calibrate them, so the apparent fit can be too optimistic. Generate scores from models that exclude their scored observations, fit the map to those out-of-fold pairs, and reserve independent data for evaluation.

Check Your Understanding

Use the definitions and arguments above to answer the following.

In $K$-fold cross-fitting, which observations are excluded when generating the score for a case in fold $k$?
Why are out-of-fold score-label pairs suitable for fitting a calibration map but not an independent evaluation sample?
What exact sample identity follows from the intercept score equation for a finite Platt maximum-likelihood fit?
Give one reason isotonic calibration can produce unstable probabilities in a small calibration sample.
State the independence condition that makes the mean evaluation loss conditionally unbiased for the deployed procedure's risk.
Why can refitting a base model on all development data after cross-fitting create a score-scale concern, even if the final evaluation set remains independent?