Cross-Validated Calibration Assessment

Why Assess Calibration?

A genomic classifier may rank individuals correctly yet give probabilities that are systematically too high or too low. Calibration concerns whether predicted risks correspond to observed event frequencies: among individuals assigned probabilities near 0.20, for example, an event should occur in about 20% of cases over repeated comparable predictions. This is distinct from discrimination, which concerns how well predictions order individuals with and without the outcome.

In cross-validation, each individual’s out-of-fold prediction is made by a model that was not trained on that individual. These predictions are the appropriate starting point for assessing performance on held-out data. Any data-dependent step, including fitting a recalibration model, must remain within the training data for each fold. The assessment outcomes in the held-out folds must not determine the predictions being assessed.

Definition: Let \(Y_i\in\{0,1\}\) be the observed outcome for individual \(i\), and let \(\hat{p}_i\) be that individual’s out-of-fold predicted probability of \(Y_i=1\). Predictions are calibrated if, for predicted probability \(p\), the event frequency among individuals assigned that probability is \(p\). In practice, calibration is assessed approximately over groups or by fitting a calibration model; finite samples will not match every predicted value exactly.

Calibration Plots and the Brier Score

A calibration plot places predicted probabilities on the horizontal axis and observed event frequencies on the vertical axis. The ideal reference is the diagonal \(y=x\). A point above the diagonal indicates that the observed frequency exceeds the average prediction in that region; points below it indicate that predictions exceed the observed frequency. A common construction groups predictions into bins, then plots each bin’s mean predicted probability against its event proportion. Bin counts should be reported because a frequency based on a few individuals is unstable.

The Brier score gives a single summary of squared probability error. For \(n\) individuals with binary outcomes, it is the mean squared difference between the predicted probabilities and the observed outcomes:

\mathrm{Brier}=\frac{1}{n}\sum_{i=1}^{n}(Y_i-\hat{p}_i)^2.

A lower Brier score is better for the evaluated predictions, but it is not a pure measure of calibration: it reflects both calibration and the ability to distinguish outcomes. Its scale depends on event prevalence. A useful reference is the Brier score from predicting the same prevalence \(\bar{Y}\) for every individual, where \(\bar{Y}=n^{-1}\sum_iY_i\). This reference score is \(\bar{Y}(1-\bar{Y})\). A model’s score below this reference is an improvement over that constant prediction on this sample, not proof that every aspect of its probability estimates is adequate.

Worked Example: A Binned Plot and Brier Score

Suppose a hypothetical cross-validation produces 20 out-of-fold predictions: 10 individuals receive \(\hat{p}=0.20\), of whom 1 has the event; the other 10 receive \(\hat{p}=0.60\), of whom 5 have the event. The outcome is binary, and each individual contributes one held-out prediction.

For the first bin, the mean predicted probability is 0.20 and the observed event frequency is \(1/10=0.10\). For the second, the mean prediction is 0.60 and the observed frequency is \(5/10=0.50\). Both plotted points fall below the diagonal: these predictions are higher than the corresponding observed frequencies in these bins.

For the Brier score, the first bin contributes one squared error of \((1-0.20)^2=0.64\) and nine errors of \((0-0.20)^2=0.04\), for a total of \(0.64+9(0.04)=1.00\). The second contributes five errors of \((1-0.60)^2=0.16\) and five of \((0-0.60)^2=0.36\), for a total of \(5(0.16)+5(0.36)=2.60\). Thus:

\mathrm{Brier}=\frac{1.00+2.60}{20}=0.180.

There are 6 events among 20 individuals, so \(\bar{Y}=0.30\). The prevalence-only reference is \(0.30(1-0.30)=0.210\). The model’s Brier score of 0.180 is lower than that reference for these predictions, while the binned plot still shows overprediction in both bins. A favourable comparison on one score does not erase a visible calibration pattern.

Calibration Intercepts and Slopes

Calibration can also be summarized with logistic regression. For probabilities strictly between 0 and 1, define the logit as \(\operatorname{logit}(p)=\log\{p/(1-p)\}\). The calibration model is

\operatorname{logit}\{P(Y_i=1)\}=a+b\,\operatorname{logit}(\hat{p}_i).

Here \(a\) is the calibration intercept and \(b\) is the calibration slope. If predictions are calibrated according to this model, the ideal values are \(a=0\) and \(b=1\). A slope below 1 commonly indicates predictions that are too extreme: the fitted relationship is less steep than the ideal line. A slope above 1 commonly indicates predictions that are not extreme enough. These are broad interpretations of the fitted relationship, not diagnoses that should replace a plot or examination of the data.

A related quantity, calibration-in-the-large, assesses overall over- or underprediction while fixing the slope at 1. It fits an intercept \(c\) in \(\operatorname{logit}\{P(Y_i=1)\}=c+\operatorname{logit}(\hat{p}_i)\). A positive \(c\) indicates that the observed event rate is higher than predicted overall; a negative \(c\) indicates that it is lower. This differs from the intercept \(a\) estimated jointly with a free slope \(b\). When the slope is not 1, the two intercepts need not agree.

Worked Example: Interpreting a Calibration Intercept and Slope

Use the 20 hypothetical predictions from the preceding example. There are two distinct prediction values. At \(\hat{p}=0.20\), the observed event proportion is 0.10; at \(\hat{p}=0.60\), it is 0.50. In this two-group setup, the fitted logistic calibration model matches the two observed proportions. The corresponding logits are:

x_1=\operatorname{logit}(0.20)=\log(0.20/0.80)=-1.3863,\qquad x_2=\operatorname{logit}(0.60)=\log(0.60/0.40)=0.4055,

z_1=\operatorname{logit}(0.10)=\log(0.10/0.90)=-2.1972,\qquad z_2=\operatorname{logit}(0.50)=0.

The line through \((x_1,z_1)\) and \((x_2,z_2)\) has slope and intercept:

b=\frac{z_2-z_1}{x_2-x_1} =\frac{0-(-2.1972)}{0.4055-(-1.3863)} =\frac{2.1972}{1.7918}\approx1.226, \qquad a=z_1-bx_1\approx-2.1972-(1.226)(-1.3863)\approx-0.497.

The estimated slope exceeds 1, consistent here with predictions that are too moderate: the observed frequency is below 0.20 in the low-prediction group and below 0.60 in the high-prediction group, so the two observed frequencies are farther apart on the logit scale than the predictions. The negative joint intercept describes this fitted line at predicted logit zero; it is not, by itself, the calibration-in-the-large estimate.

Derivation and Recalibration Without Leakage

The calibration model follows from logistic regression applied to the outcome using the logit of the predicted probability as a predictor. If the relationship is ideal, substituting \(a=0\) and \(b=1\) gives \(\operatorname{logit}\{P(Y_i=1)\}=\operatorname{logit}(\hat{p}_i)\), and applying the inverse logit gives \(P(Y_i=1)=\hat{p}_i\). The reference values therefore express the definition of calibration on the log-odds scale.

The same model can produce recalibrated probabilities, \(\hat{p}^{\,*}=\operatorname{expit}\{a+b\operatorname{logit}(\hat{p})\}\), where \(\operatorname{expit}(u)=1/(1+e^{-u})\). But estimating \(a\) and \(b\) on the outcomes later used to assess \(\hat{p}^{\,*}\) makes the assessment optimistic: those outcomes have already influenced the predictions. Instead, estimate recalibration parameters using only the training data in each fold, then apply that transformation to the corresponding held-out predictions. Assess the transformed out-of-fold predictions against their held-out outcomes.

Worked Example: Applying a Fold-Local Recalibration

In a hypothetical outer fold, suppose a recalibration model is fitted using only that fold’s training data and yields \(a=-0.40\), \(b=1.20\). A held-out individual has a base-model prediction of \(\hat{p}=0.25\). Transform this prediction using the training-fold estimates:

\operatorname{logit}(0.25)=\log(0.25/0.75)=-1.0986, \qquad a+b\operatorname{logit}(0.25)=-0.40+1.20(-1.0986)=-1.7183.

\hat{p}^{\,*}=\operatorname{expit}(-1.7183) =\frac{1}{1+e^{1.7183}}\approx0.152.

The held-out outcome is not used to estimate either coefficient or transform the probability. It is used only afterward, alongside the other held-out outcomes, to assess the recalibrated predictions. Repeating this procedure across folds gives an honest cross-validated assessment of the specified recalibration procedure.

Diagnostics, Alternatives, and Limitations

A calibration plot, Brier score, and logistic calibration model answer related but different questions. The plot can reveal local departures that a single number hides, but its appearance depends on binning and sample size. The Brier score summarizes probability error but mixes calibration and discrimination. Intercept and slope give compact summaries on the log-odds scale but may conceal nonlinear patterns. Read them together rather than treating any one as a complete calibration verdict.

Check the number of events and non-events in each bin, and avoid interpreting noisy frequencies from sparse groups as stable patterns. Compare alternative bin widths or use a smooth calibration curve as a diagnostic, while recognizing that smoothing choices affect the display. Very extreme predictions can have substantial influence on logistic calibration estimates. If probabilities equal exactly 0 or 1, their logits are infinite; a prespecified clipping rule may be needed before fitting a logit-based calibration model. Such a rule must not be chosen using held-out outcomes.

A common mistake is to fit one calibration model to all out-of-fold predictions and then report the calibration of its fitted values on those same outcomes as if it were an independent assessment. The fitted values have been adapted to those outcomes, so this reuses evaluation information. Use the pooled out-of-fold predictions to describe the original model’s calibration; if recalibration is part of the procedure, fit it within training folds and assess only its held-out transformed predictions.

Key takeaway: Calibration assessment asks whether out-of-fold probabilities agree with observed frequencies. Keep assessment outcomes out of every step that constructs or recalibrates the predictions being assessed.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Among 40 individuals assigned predictions near 0.10, 8 experience the event. What are the observed event frequency and the direction of departure from ideal calibration?
For 5 outcomes \(Y=(1,0,1,0,0)\) and predictions \((0.8,0.2,0.6,0.4,0.1)\), calculate the Brier score.
What do the ideal calibration intercept and slope equal, and what does a slope below 1 commonly suggest?
Why should recalibration parameters be fitted separately within training folds rather than on all assessment outcomes?

Solutions. (1) The event frequency is \(8/40=0.20\), above the prediction of about 0.10; this group is underpredicted. (2) The squared errors are \(0.04,0.04,0.16,0.16,0.01\), so the Brier score is \(0.41/5=0.082\). (3) The ideal values are \(a=0\) and \(b=1\); a slope below 1 commonly suggests predictions that are too extreme. (4) Fitting on assessment outcomes lets those outcomes influence the probabilities whose performance is being reported, causing optimistic evaluation.