Comparing Calibration and Discrimination

Two Different Questions About Predicted Probabilities

A supervised integration strategy can combine several omics blocks to predict a binary outcome, such as whether a sample belongs to a disease group. Its output may be a probability for each sample. Comparing these predictions involves at least two distinct questions: are the probabilities numerically trustworthy, and do they rank cases above non-cases? These are calibration and discrimination, respectively. A strategy can perform well on one and poorly on the other, so a single overall ranking can conceal an important difference.

The summaries below assume a binary outcome \(Y_i\in\{0,1\}\) and a predicted probability \(\hat{p}_i\in[0,1]\) for each of \(n\) independent evaluation units. Here \(Y_i=1\) denotes the event of interest. Predictions should be evaluated on data not used to fit or tune the corresponding model; otherwise, the summaries may reflect fitting performance rather than performance on new units. When comparing integration strategies, use the same evaluation units and outcome definition wherever possible.

Definition: A prediction system is calibrated when, among units assigned predicted probability \(q\), the event occurs with frequency \(q\), in the population and at the relevant prediction resolution. Formally, ideal calibration means \(P(Y=1\mid \hat{p}=q)=q\) for probability values \(q\) that occur. Discrimination is the ability to assign higher predicted probabilities to event cases than to non-event cases. Calibration concerns probability scale; discrimination concerns ordering.

In finite validation samples, calibration is estimated rather than known. A model can have a correct average predicted probability yet be miscalibrated at particular risk levels. Conversely, a calibration plot may look close to the diagonal simply because the sample is too small to reveal departures. Interpret summaries with their resolution and uncertainty in mind.

Calibration Summaries

A direct empirical calibration summary groups predictions into bins. For bin \(b\), let \(n_b\) be the number of units, \(\bar{p}_b\) their mean predicted probability, and \(\bar{y}_b\) their observed event proportion. A calibration plot places \(\bar{p}_b\) on the horizontal axis and \(\bar{y}_b\) on the vertical axis. Perfect agreement would place the points on the diagonal. The expected calibration error for a specified binning is

\mathrm{ECE}=\sum_{b=1}^{B}\frac{n_b}{n}\left|\bar{y}_b-\bar{p}_b\right|.

ECE is a descriptive summary for those particular bins, not a bin-free measure of population calibration. Wide bins can conceal errors that go in opposite directions within a bin; narrow bins can be unstable when they contain few units. Always report or explain the binning choice when using ECE.

Calibration-in-the-large asks whether average predicted risk agrees with the overall event frequency. A simple descriptive difference is \(\bar{p}-\bar{y}\), where \(\bar{p}\) is the mean prediction and \(\bar{y}\) is the observed event proportion. For a more model-based assessment, the calibration intercept and slope can be estimated by fitting

\operatorname{logit}P(Y_i=1)=a+b\,\operatorname{logit}(\hat{p}_i), \qquad \operatorname{logit}(u)=\log\left(\frac{u}{1-u}\right).

Ideal values are \(a=0\) and \(b=1\). The intercept assesses systematic over- or underprediction when the slope is fixed at 1; the slope assesses whether predictions are too extreme or too compressed. These regression summaries need enough events and non-events, and predictions strictly between 0 and 1 for the displayed logit. Estimated values can be highly uncertain in small samples.

Discrimination and Overall Probability Scores

The area under the receiver operating characteristic curve (AUC) has a pairwise interpretation. It is the probability that a randomly selected event case receives a higher prediction than a randomly selected non-event case, counting a tie as one-half. Its sample estimate is

\widehat{\mathrm{AUC}} = \frac{\text{number of correctly ordered case--non-case pairs} +\tfrac12(\text{number of tied pairs})} {n_1n_0},

where \(n_1\) and \(n_0\) are the numbers of event and non-event units. AUC is a ranking summary: it does not require predicted probabilities to match observed event frequencies. AUC near 0.5 indicates ranking close to chance in this pairwise sense; it does not establish that predictions are calibrated.

The Brier score measures squared probability error:

\mathrm{Brier}=\frac{1}{n}\sum_{i=1}^{n}(\hat{p}_i-Y_i)^2.

Lower values are better. Unlike AUC, Brier score uses the numerical probabilities, so it penalizes confident incorrect predictions. It is an overall probability score, not a pure calibration measure: it reflects both calibration and the ability to separate units with different outcomes. Its scale depends on event prevalence and the prediction problem, so comparisons are most interpretable when strategies are evaluated on the same units and outcome.

Worked Example: AUC With Tied Predictions

Suppose a hypothetical validation set contains three event cases and two non-event cases. Their predictions are \(0.8, 0.5, 0.4\) for the cases and \(0.7, 0.5\) for the non-cases. The AUC denominator is \(n_1n_0=3\times2=6\) pairs.

The case with prediction \(0.8\) outranks both non-cases, contributing 2 wins. The case with prediction \(0.5\) loses to \(0.7\) and ties \(0.5\), contributing \(0.5\). The case with prediction \(0.4\) loses to both, contributing 0. Thus

\widehat{\mathrm{AUC}}=\frac{2+0.5+0}{6} =\frac{2.5}{6}=0.4167.

This small sample gives an AUC below 0.5: the observed ordering is worse than chance by the pairwise definition. The estimate is very uncertain with only six pairs, and should not be treated as evidence that the model will reverse ranks in a population.

Why the Rankings Can Disagree

A strictly increasing transformation of predictions preserves their ordering, and therefore preserves AUC, provided ties and ordering are unchanged. It can nevertheless alter probability errors and calibration summaries. This is why a ranking metric and a probability-accuracy metric can favour different versions of a model. The following justification makes the distinction explicit.

Consider a monotone transformation.
Let \(g\) be strictly increasing and replace each \(\hat{p}_i\) by \(g(\hat{p}_i)\), with transformed values still in \([0,1]\).

Check every pairwise comparison.
For any two units \(i,j\), \(\hat{p}_i>\hat{p}_j\) if and only if \(g(\hat{p}_i)>g(\hat{p}_j)\). Thus the correctly ordered and tied case--non-case pairs do not change.

Compare probability errors.
The transformed errors are \(g(\hat{p}_i)-Y_i\), which need not equal the original errors. Therefore the Brier score and calibration summaries can change even though AUC does not.

This argument concerns the ranking property only; it does not guarantee that a transformation improves calibration. A transformation must be assessed on evaluation data separate from the data used to choose it.

Worked Example: Strong Ranking, Different Calibration Summary

Consider eight hypothetical validation units: four event cases and four non-cases. Strategy A predicts \(0.9,0.8,0.7,0.6\) for the cases and \(0.54,0.53,0.52,0.51\) for the non-cases. Strategy B predicts \(0.5\) for every unit. Use two fixed bins, \([0,0.5)\) and \([0.5,1]\).

For Strategy A, every case prediction exceeds every non-case prediction, so \(\widehat{\mathrm{AUC}}=16/16=1\). For Strategy B, every case--non-case pair is tied, so \(\widehat{\mathrm{AUC}}=(0.5\times16)/16=0.5\).

All eight predictions from each strategy fall in the upper bin. Its observed event proportion is \(4/8=0.5\). Strategy A's mean prediction is

\bar{p}_A=\frac{0.9+0.8+0.7+0.6+0.54+0.53+0.52+0.51}{8} =\frac{5.10}{8}=0.6375.

Thus its ECE for these bins is \(|0.5-0.6375|=0.1375\). Strategy B has mean prediction \(0.5\), so its ECE is \(|0.5-0.5|=0\). On these summaries, A ranks cases perfectly but has worse binned calibration; B has no discrimination but matches the observed frequency in the sole occupied bin.

The Brier scores add a useful qualification. For A, the sum of squared errors for cases is \(0.1^2+0.2^2+0.3^2+0.4^2=0.30\), and for non-cases it is \(0.54^2+0.53^2+0.52^2+0.51^2=1.103\). Therefore its Brier score is \((0.30+1.103)/8=0.1754\). For B, every squared error is \(0.25\), so its Brier score is \(8(0.25)/8=0.25\). B has better ECE for the chosen bins, while A has the lower Brier score. These sample summaries do not prove that B is calibrated in the population: its predictions have no variation, and the binning cannot assess calibration at finer risk levels.

Worked Example: Recalibration Can Improve Probability Error Without Changing AUC

In a hypothetical validation set, two cases have predictions \(0.9\) and \(0.8\), and two non-cases have predictions \(0.4\) and \(0.3\). Every case outranks every non-case, so the AUC is \(4/4=1\). The mean prediction is \((0.9+0.8+0.4+0.3)/4=0.6\), while the observed event proportion is \(2/4=0.5\).

Consider subtracting \(0.1\) from each prediction. The transformed predictions are \(0.8,0.7\) for cases and \(0.3,0.2\) for non-cases. They remain in \([0,1]\), and their ordering is unchanged, so AUC remains 1. The transformed mean is \(2.0/4=0.5\), matching the observed proportion in this sample.

Before transformation, the sum of squared errors is \(0.1^2+0.2^2+0.4^2+0.3^2=0.30\), giving Brier score \(0.30/4=0.075\). Afterwards it is \(0.2^2+0.3^2+0.3^2+0.2^2=0.26\), giving \(0.26/4=0.065\). This transformation improves the sample Brier score and average prediction alignment here, but the tiny sample does not establish general calibration. Subtracting a constant is not a universally valid recalibration rule; outside this example it could even produce invalid probabilities.

Assumptions, Diagnostics, and Alternatives

Metrics are only as meaningful as the evaluation design and the prediction target. Check that the outcome is coded consistently, each unit contributes an appropriate independent evaluation observation, and the same target population and evaluation units are used for each strategy. If multiple samples come from the same person, treating them as independent can make uncertainty appear smaller than it is. Uncertainty intervals for differences should account for the paired predictions on the same units; resampling evaluation units in a paired fashion is one practical approach, with clusters resampled together when the independent unit is a cluster.

Useful diagnostics include a calibration plot with bin counts, the observed event prevalence, the distribution of predicted probabilities, and a ROC curve or AUC. For rare outcomes, also consider precision-recall summaries: the ROC curve can look favourable when false positives are numerous relative to the small number of cases. Threshold-specific sensitivity and specificity answer a decision question at a chosen cut-off, not the general question of ranking or probability calibration. A threshold should be selected for a stated use, not chosen after inspecting the evaluation outcomes.

The main alternatives answer different questions. AUC assesses pairwise ordering; Brier score assesses squared probability error; log loss, \(-n^{-1}\sum_i[Y_i\log(\hat{p}_i)+(1-Y_i)\log(1-\hat{p}_i)]\), penalizes confident errors especially strongly; and calibration plots or regression summaries inspect agreement between probabilities and frequencies. No one summary replaces the others. Report a small set aligned to the intended prediction use rather than selecting whichever metric makes a strategy look best.

When integration strategies rank differently, first verify that they were evaluated on the same outcomes and units and that tuning did not use the evaluation results. Then state which property each metric measures and report its estimate with uncertainty where feasible. A calibration advantage does not imply better discrimination, and higher AUC does not imply more trustworthy risk probabilities. Neither association of an omics block with the outcome nor a predictive gain establishes a causal effect.

Common Pitfall

Wrong reasoning: “Strategy A has the highest AUC, so its probabilities are the most accurate.” AUC only assesses ordering; any strictly increasing transformation preserves that ordering.
Correction: Assess probability agreement separately using a calibration summary and a proper probability score such as the Brier score or log loss. Inspect the calibration plot and its bin counts, and interpret estimates in light of evaluation-sample size and uncertainty.
Wrong reasoning: “ECE is zero, so the model is calibrated.” ECE depends on the chosen bins and can hide within-bin departures.
Correction: Treat ECE as a bin-specific summary, report the binning, and use plots and additional summaries to diagnose limitations.

Key takeaway: Calibration asks whether predicted probabilities agree with event frequencies; discrimination asks whether event cases receive higher predictions than non-cases. Compare both properties because integration strategies can rank differently on each.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A model's predictions are transformed by a strictly increasing function that keeps them within \([0,1]\). Which of AUC and Brier score must stay unchanged, and why?
In a bin of 20 units, 6 events occur and the mean predicted probability is \(0.40\). What is the bin's absolute calibration gap?
A dataset has 3 event cases and 4 non-cases. A case prediction outranks all four non-cases, a second case ties one non-case and outranks the other three, and the third case ranks below every non-case. What is the AUC?
Why can a low ECE fail to establish calibration?

Solutions. (1) AUC stays unchanged because every pairwise ordering and tie is preserved. Brier score need not stay unchanged because transformed probabilities can have different squared errors. (2) The observed event proportion is \(6/20=0.30\), so the absolute gap is \(|0.30-0.40|=0.10\), or 10 percentage points. (3) The pairwise contribution is \(4+3.5+0=7.5\) wins out of \(3\times4=12\) pairs, so AUC is \(7.5/12=0.625\). (4) ECE depends on binning; errors within a bin can be hidden, and small bins may be unstable. It is a descriptive sample summary, not proof of population calibration.