Why calibration matters across populations
A classifier may rank higher-risk patients above lower-risk patients yet systematically overstate or understate their probabilities of disease. That distinction matters when a predicted probability informs screening, treatment discussions, or resource planning: a ranking can be useful while the numerical risks remain unreliable. Calibration assessment asks whether predicted probabilities agree with outcome frequencies in the population where the model will be used.
For a multi-omics prediction model, calibration should be evaluated using predictions produced by a fixed pipeline on independent evaluation subjects from the target population. The model, preprocessing, and any probability transformation must be fixed before examining evaluation outcomes. The earlier material on independent units and held-out test sets is relevant here: repeated measurements or related subjects cannot be treated as independent observations, and repeated tuning against a test set compromises its role as an independent assessment.
Calibration is population-specific. A model calibrated in its development cohort need not be calibrated in an independent population with a different event prevalence, clinical pathway, assay distribution, or case mix. Such differences can affect observed risk even if the model's ability to rank subjects is largely retained. The target population, outcome definition, prediction time, and follow-up window therefore need to be specified before interpreting calibration.
Calibration plots and grouped summaries
A calibration plot places predicted risk on the horizontal axis and observed event frequency on the vertical axis. The ideal reference is the diagonal \(y=x\): for example, predictions near 0.30 should correspond to an event frequency near 0.30. A curve above the diagonal indicates that observed risks tend to exceed predictions in that region (underprediction); a curve below it indicates overprediction.
Because each evaluation subject has a binary outcome, a plot usually estimates observed risk by grouping predictions or smoothing across them. For a group \(g\) containing \(n_g\) subjects, a simple point is \((\bar{p}_g,\bar{y}_g)\), where \(\bar{p}_g\) is the group's mean predicted probability and \(\bar{y}_g\) its observed event proportion. Grouping is easy to interpret but depends on bin boundaries and can hide local departures. A smooth curve avoids fixed bins but depends on the smoothing method and can be unstable where data are sparse. Displaying group sizes and uncertainty intervals helps show how much evidence each part of the curve contains.
Worked Example: Aggregate agreement can hide local errors
Suppose an independent evaluation set has four prediction groups. Each row gives the number of subjects, mean predicted risk, and observed event proportion. These are hypothetical data.
| Group size | Mean predicted risk | Observed event proportion |
|---|---|---|
| 50 | 0.10 | 0.10 |
| 40 | 0.30 | 0.30 |
| 40 | 0.60 | 0.60 |
| 70 | 0.80 | 0.80 |
The total predicted number of events, expressed as the sum of predicted probabilities, is
Thus the mean predicted risk is \(97/200=0.485\). The observed event counts are \(5,12,24,\) and \(56\), also totaling 97, so the overall observed event proportion is \(97/200=0.485\). Yet this exact aggregate agreement does not establish calibration at every risk level: errors in different groups could cancel. In these particular hypothetical groups, the observed proportions equal the group mean predictions, but the finite counts still leave sampling uncertainty, and the group summaries do not establish calibration for every individual prediction value.
Calibration intercept and slope
A plot gives a flexible visual summary. Two logistic regression summaries provide compact diagnostics. They require binary outcomes and probabilities strictly between zero and one so that their logits are finite. They are estimated in the evaluation population; their interpretation depends on that population and does not prove that the entire calibration curve has a particular shape.
The calibration-in-the-large intercept \(a\) is estimated from a logistic regression with the logit of each model prediction as a fixed offset:
Here \(p_i\) is the model's predicted probability for subject \(i\). The ideal intercept is \(a=0\). A positive intercept indicates that the observed event odds are higher than predicted overall, conditional on this offset formulation; a negative intercept indicates lower observed odds. This diagnostic shifts the prediction scale without estimating a new slope.
The calibration slope is estimated by fitting both an intercept and a coefficient:
Ideal values are \(a=0\) and \(b=1\). A slope below 1 often signals predictions that are too extreme: high predictions are too high and low predictions too low, relative to the fitted logistic relationship. A slope above 1 indicates predictions that are insufficiently spread under that relationship. The intercept in this joint fit is not generally identical to the calibration-in-the-large intercept, because the slope is also estimated.
Worked Example: Interpreting an intercept and slope
Assume a hypothetical independent evaluation set yields a calibration-in-the-large estimate \(a=0.40\), and a separate joint logistic recalibration fit yields \(a=-0.05\) and \(b=0.72\). Treat these as fitted estimates from the stated models, not as values that can be recovered from summary counts alone.
In the offset model, a 0.40 intercept multiplies predicted odds by \(\exp(0.40)\approx1.492\). For a prediction of 0.20, the predicted odds are \(0.20/0.80=0.25\); the fitted shifted odds are \(0.25(1.492)=0.373\), corresponding to probability \(0.373/(1+0.373)\approx0.272\). This positive intercept indicates overall underprediction on the odds scale under the offset model.
For the joint fit, a prediction \(p=0.80\) has logit \(\log(0.80/0.20)=\log(4)\approx1.386\). Its fitted log odds are \(-0.05+0.72(1.386)=0.948\), giving probability \(1/(1+\exp(-0.948))\approx0.721\). The slope below 1 compresses predictions toward the middle on the log-odds scale. The joint intercept and slope summarize a fitted logistic relationship; neither guarantees adequate calibration at every risk value.
A short justification for the aggregate calibration check
If the defining calibration condition holds, the mean outcome and mean prediction agree in the population. The justification uses the law of iterated expectation:
For binary \(Y\), \(E(Y\mid P)=P(Y=1\mid P)\), which equals \(P\) under calibration. This proves that perfect calibration implies zero average prediction error. The converse does not hold: positive and negative conditional errors can cancel in other examples; the displayed groups have no group-level errors to cancel. Therefore, a correct overall mean is a useful check but not a sufficient calibration assessment.
Proper scores and discrimination are different questions
The Brier score and log loss summarize the quality of probabilistic predictions using both outcomes and predicted probabilities. For \(n\) evaluation subjects, their standard forms are
Both are proper scoring rules: in expectation, a subject's score is optimized by reporting the true event probability, conditional on the information available. Lower average scores are better for the same outcome definition and comparable evaluation population. Log loss penalizes confident incorrect predictions particularly strongly; predictions of exactly zero or one can produce infinite loss when contradicted by the outcome. Neither score, by itself, isolates calibration: it reflects probability accuracy overall, including aspects related to separating subjects with different outcomes.
Worked Example: Calculating two proper scores
Consider four hypothetical independent subjects with predictions \(0.10,0.40,0.60,0.80\) and outcomes \(0,0,1,1\), respectively. The Brier score is
The log loss is
These are dimensionless average scores for this four-subject set. They describe the predictions and outcomes jointly; they do not establish that the model is calibrated across risk levels. A calibration plot and intercept/slope diagnostics address that question more directly.
Discrimination concerns how well a model separates or ranks subjects who experience the event from those who do not. Calibration concerns numerical agreement between risks and event frequencies. A model can rank subjects well but overpredict risk across the board; it can also be calibrated on average while ranking subjects poorly. A strictly increasing probability recalibration can change numerical risks and proper scores while leaving the ordering, and thus rank-based discrimination, unchanged. This is why probability calibration should not be inferred from discrimination metrics alone, or vice versa.
| Question | Typical assessment | What it does not establish by itself |
|---|---|---|
| Do predicted risks match observed frequencies? | Calibration plot, intercept, slope | Good ranking of cases above non-cases |
| Are probability forecasts accurate overall? | Brier score or log loss | Calibration alone, separated from other prediction properties |
| Are cases generally ranked above non-cases? | Discrimination measures | Accurate numerical risk probabilities |
Assumptions, diagnostics, and limitations
Calibration findings are only as meaningful as the evaluation design. The outcome must represent the intended event over a defined time window, and predicted probabilities must refer to that same event and time. Subjects should represent the intended target population, or any sampling differences need an appropriate design-based adjustment. If outcomes are observed through unequal follow-up, missingness, or a sampling scheme that changes event prevalence, unadjusted frequencies and scores may not estimate target-population performance. Dependence among subjects or repeated observations also requires an analysis that respects the independent sampling units.
Inspect the number of events and non-events across the prediction range, group sizes, uncertainty intervals, and whether the curve changes substantially under reasonable grouping or smoothing choices. Sparse extremes can make plots and slope estimates unstable. A very large evaluation set can reveal small departures from the diagonal that may have little practical consequence; a small set can conceal important departures. Estimates of intercept and slope should therefore be interpreted with uncertainty and alongside the plot, not as pass/fail thresholds.
If systematic miscalibration is found, intercept-only adjustment or logistic recalibration may be considered using data appropriate for updating the model. The recalibrated model then needs independent evaluation; fitting and reporting the adjustment on the same outcomes gives an optimistic assessment. Recalibration changes predicted probabilities, not the biological or causal meaning of predictors, and cannot guarantee performance in a further population. When the target population changes again, calibration may change again.
Common pitfall
- Wrong reasoning: “The average predicted risk equals the observed event proportion, so the model is calibrated.”
- Correction: Equality of the averages verifies only a global condition. Conditional overprediction in one risk range can cancel underprediction elsewhere. Examine the calibration curve and use intercept and slope as complementary summaries.
- Wrong reasoning: “The classifier discriminates well, so its risk estimates are trustworthy.”
- Correction: Ranking and probability agreement are distinct properties. Assess both when decisions use numerical risks.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A model's mean predicted risk is 0.30 and the evaluation event proportion is 0.30. Does this prove calibration? Why?
- A calibration plot lies mostly below the diagonal. What does that suggest about predictions relative to observed risk?
- For predictions of 0.70 and 0.30, respectively, outcomes are 1 and 0. Calculate the Brier score.
- A joint logistic calibration fit has slope \(b=0.60\). What does this suggest, and what should not be concluded from it alone?
- Can a model retain its subject ranking after a monotone recalibration while its predicted risks change? Explain.
Solutions. (1) No; errors in different risk ranges can cancel, so inspect conditional calibration. (2) Observed risks tend to be lower than predicted, indicating overprediction. (3) \(\{(0.70-1)^2+(0.30-0)^2\}/2=(0.09+0.09)/2=0.09\). (4) The predictions may be too extreme under the fitted logistic relationship; the slope alone does not describe every local departure or prove overall poor discrimination. (5) Yes. A strictly increasing transformation preserves order but changes numerical probabilities, so discrimination by ranking may remain while calibration and proper scores change.