Calibration Diagnostics for Predicted Binary Risks

Why Use More Than One Calibration Diagnostic?

A binary risk prediction is a probability \(p_i\) assigned to an observation whose outcome \(Y_i\) is 0 or 1. Calibration asks whether predictions agree with observed event frequencies. A useful assessment does not reduce this question to one number: agreement in the overall event rate can coexist with serious errors at particular predicted risks, while a plot can look persuasive even when its estimates are imprecise.

Three complementary views help: a calibration plot shows how observed risk varies across predicted risk; calibration-in-the-large summarizes overall over- or underprediction; and a calibration slope assesses whether predictions vary too much or too little. These are diagnostics, not interchangeable tests. Their interpretation depends on how predictions were obtained, the validation sample, and the distribution of predicted risks.

In a validation sample, the predictions \(p_i\) are treated as given and compared with outcomes \(Y_i\). The observed data cannot establish the conditional equality at every possible value of \(P\); it can only provide estimates, with uncertainty. A calibration plot estimates the relationship between outcome and prediction. Its ideal reference is the diagonal \(y=p\), but uncertainty and the amount of data near each risk value matter as much as visual proximity.

Three Complementary Summaries

A calibration plot can be constructed by grouping predictions into bins and plotting each bin's mean observed outcome against its mean predicted risk, or by estimating a smooth calibration curve. Binned plots are simple, but the result depends on the binning and can hide within-bin variation. Smooth curves avoid abrupt bin boundaries but rely on a smoothing method and can be uncertain, especially where few predictions occur. Uncertainty bands should be read as uncertainty about the estimated curve, not as proof of calibration wherever the diagonal falls inside a band.

Two logistic recalibration summaries provide compact complements. Write \(z_i=\operatorname{logit}(p_i)=\log\{p_i/(1-p_i)\}\), requiring \(0<p_i<1\). Calibration-in-the-large fits an intercept while using \(z_i\) as a fixed offset. The calibration slope fits both an intercept and a coefficient for \(z_i\). In both cases, the outcome is modeled as Bernoulli; these regressions describe the validation data and do not by themselves repair the original predictions.

A positive calibration-in-the-large intercept indicates that the validation outcomes require a common upward shift on the log-odds scale when the original predictions are used as offsets; a negative intercept indicates a downward shift. It is not simply the difference between the observed and predicted event proportions. A slope below 1 typically indicates that the spread of predicted risks is too extreme relative to outcomes in this validation sample; a slope above 1 indicates that the spread is too compressed. The fitted intercept in the slope model can differ from the calibration-in-the-large intercept, because the slope model also changes the contribution of each predicted risk.

What Average Agreement Does—and Does Not—Show

Calibration implies agreement in average risk, but the reverse is not true. The distinction is particularly important when predictions vary substantially. A model can predict too low for one set of observations and too high for another, with the errors cancelling in the total number of events.

Proof. By the law of iterated expectation,

For the converse, let \(P\) equal 0.1 or 0.9 with probability one-half each, and let the conditional event probability be 0.9 when \(P=0.1\) and 0.1 when \(P=0.9\). Then \(E[P]=(0.1+0.9)/2=0.5\) and \(E[Y]=(0.9+0.1)/2=0.5\), but \(E[Y\mid P]\ne P\) at either prediction value. Thus equal means can conceal substantial miscalibration. \(\square\)

Why the Recalibration Fits Have Their Score Equations

The logistic summaries are tied to the validation data through likelihood equations. Stating these equations makes clear what the fitted intercept and slope actually enforce. Assume that, conditional on the fixed predictions, \(Y_1,\ldots,Y_n\) are independent Bernoulli outcomes, and consider finite maximum-likelihood estimates in the stated model.

Derivation. Ignoring constants that do not depend on the parameters, the slope-model log-likelihood is

Differentiating gives \(\partial\ell/\partial\alpha=\sum_i(Y_i-q_i)\) and \(\partial\ell/\partial\beta=\sum_i z_i(Y_i-q_i)\), because the derivative of \(\log\{1+\exp(t)\}\) is \(\operatorname{expit}(t)\). At a finite interior maximum, both derivatives are zero, which proves the two equations. In the offset model, \(\beta\) is fixed at 1 and only \(\alpha\) is estimated. Differentiating with respect to \(\alpha\) gives the single equation \(\sum_i(Y_i-q_i)=0\). Thus the fitted recalibrated probabilities match the observed event total within each fitted model; they need not match each observation or each risk region. \(\square\)

This derivation also explains why the calibration-in-the-large intercept is not the raw mean difference. The fitted probabilities are \(\operatorname{expit}(z_i+\hat\alpha)\), not \(p_i+\hat\alpha\); the logit shift changes each risk nonlinearly. The score equation matches their sum to the observed number of events, while the estimated shift depends on the full set of predicted risks.

Combining Diagnostics in Practice

A disciplined assessment uses the diagnostics to answer related but distinct questions. First inspect the distribution of predictions and the number of events: a curve in a region with little data is necessarily uncertain. Then examine a calibration plot with uncertainty, checking for systematic departures from the diagonal rather than treating each bin as a definitive test. Report calibration-in-the-large to describe the overall shift and the slope to summarize whether the risk spread appears too extreme or too compressed.

These summaries are most interpretable when predictions are evaluated on data not used to fit or tune the prediction model. If the same outcomes are used both to construct predictions and assess them, apparent calibration can be optimistic. Also report the validation sample size and event count: with rare outcomes, a small change in events can substantially alter observed rates and fitted recalibration coefficients. A numerical estimate without its uncertainty and data context is easy to overinterpret.

Common Pitfalls

• Equating matching totals with calibration. The reasoning “observed and predicted event counts are close, so predictions are calibrated” is wrong when errors can cancel across risk levels. Check the calibration curve and slope as well as the overall shift.
• Reading the intercept as a risk difference. Calibration-in-the-large is a log-odds shift in an offset model, not observed mean risk minus predicted mean risk. Report the scale and, when helpful, both the fitted intercept and the raw mean difference.
• Treating the slope as a complete curve. A single slope summarizes a particular logistic recalibration model. A nonlinear departure can remain even if the slope is near 1; compare it with the plot.
• Overinterpreting bins or a smooth curve. Bin choices affect binned summaries, and smoothing affects smooth estimates. Show uncertainty and be cautious where observations are sparse.
• Attempting a slope estimate without prediction variation. If all \(p_i\) are equal, all \(z_i\) are equal, so intercept and slope cannot both be identified separately.