What Calibration Assesses
A dose-attainment probability is a forecast of how often a specified event should occur in a defined population and setting. For example, a model might predict the probability that a patient reaches an efficacy target at a candidate dose, or the probability of a toxicity event at that dose. Calibration asks whether events occur at approximately the rates those probabilities forecast. It is a property of predictions considered across cases; it does not say that a particular patient's outcome is certain or that a dose is clinically worthwhile.
The event and prediction target must be defined before assessing calibration. Let \(Y_i=1\) if case \(i\) experiences the event of interest and \(Y_i=0\) otherwise. Let \(q_i\) be the predicted probability of that same event for that case, with \(0\leq q_i\leq 1\). A case may represent a patient, a patient-dose scenario, or a simulated replicate, but the unit must be clear. For toxicity, for instance, \(Y_i=1\) should mean the specified toxicity event occurred; it should not be silently recoded as “attainment” of a favorable target.
In finite data, exact agreement is not expected. Calibration is assessed by comparing forecasts with observed frequencies while accounting for sampling or simulation variability. A calibration assessment is tied to the population, dose range, event definition, follow-up period, and prediction process used to create the \(q_i\). A good result in one setting does not establish calibration in another.
Calibration Plots and Bins
A calibration plot summarizes the relationship between predicted probabilities and observed event frequencies. The ideal reference is the diagonal line: a group with mean prediction 0.30 should have an event frequency near 0.30. To construct a binned plot, sort cases by \(q_i\), divide them into groups, and calculate for each group the mean prediction and the observed event proportion. Plot the observed proportion against the mean prediction. The group sizes should be displayed or reported because an observed frequency based on a few cases is much less precise than one based on many cases.
Bins can have equal numbers of cases (quantile bins) or equal probability widths. Equal-count bins typically stabilize the number of observations per point, but their probability ranges can differ substantially. Equal-width bins keep ranges interpretable but can leave some bins nearly empty. Binning is a descriptive summary, not a way to make the individual predictions independent or to remove uncertainty. Results can change when bin boundaries change, so report the binning rule and inspect whether the broad pattern persists under reasonable alternatives.
Worked Example: Reading a Binned Calibration Plot
Suppose a hypothetical validation set contains 200 independent patient-dose cases. The prediction for each case and its binary event outcome are available. Cases are sorted into four bins of 50. The table gives each bin’s mean predicted event probability and number of observed events.
| Bin | Mean predicted probability | Events / cases | Observed proportion |
|---|---|---|---|
| 1 | 0.10 | 6 / 50 | 0.12 |
| 2 | 0.30 | 14 / 50 | 0.28 |
| 3 | 0.60 | 32 / 50 | 0.64 |
| 4 | 0.80 | 39 / 50 | 0.78 |
For bin 3, the observed proportion is \(32/50=0.64\), compared with a mean forecast of 0.60. Its plotted point is \((0.60,0.64)\), slightly above the diagonal: events were somewhat more frequent than predicted in this bin. For bin 4, \(39/50=0.78\), compared with 0.80, so its point is slightly below the diagonal.
Across all bins, the total number of events is \(6+14+32+39=91\), giving an observed event rate of \(91/200=0.455\). The mean prediction is \((50(0.10)+50(0.30)+50(0.60)+50(0.80))/200=90/200=0.450\). Thus, overall observed and predicted rates differ by \(0.455-0.450=0.005\), or 0.5 percentage points. That close overall agreement does not erase the bin-specific departures; positive and negative deviations can offset one another.
A bin’s observed proportion is itself uncertain. A rough binomial standard error for bin 3 is \(\sqrt{0.64(1-0.64)/50}=0.0679\), or about 6.8 percentage points. This illustrates why a small visual deviation from the diagonal is not, by itself, compelling evidence of miscalibration. For formal intervals, use an appropriate binomial interval, and account for clustering if cases are not independent.
Summary Measures and What They Do Not Say
A simple overall summary is the difference between the observed event rate and the mean predicted probability: \(\bar{Y}-\bar{q}\), where \(\bar{Y}=n^{-1}\sum_iY_i\) and \(\bar{q}=n^{-1}\sum_iq_i\). A positive value means events occurred more often than forecast overall; a negative value means they occurred less often. This is a useful descriptive calibration-in-the-large check, but it can conceal errors that vary across the probability range.
The Brier score is the mean squared error of the probability forecasts for binary outcomes:
A smaller Brier score is better for the particular set of forecasts and outcomes being evaluated. It is not a pure calibration measure: it reflects both calibration and the ability to distinguish cases with different outcome risks, and it depends on event prevalence. Therefore, a Brier score should not replace a calibration plot or be interpreted alone as proof of calibration.
Worked Example: Overall Calibration Difference and Brier Score
In a hypothetical five-case validation set, predictions are \(q=(0.20,0.40,0.60,0.80,0.70)\), and outcomes are \(Y=(0,1,1,0,1)\). The observed event rate is \(\bar{Y}=3/5=0.60\). The mean prediction is \((0.20+0.40+0.60+0.80+0.70)/5=2.70/5=0.54\). The overall difference is \(0.60-0.54=0.06\): events were six percentage points more frequent than the average forecast.
The squared errors are \(0.20^2=0.04\), \((0.40-1)^2=0.36\), \((0.60-1)^2=0.16\), \(0.80^2=0.64\), and \((0.70-1)^2=0.09\). Therefore, \(\mathrm{BS}=(0.04+0.36+0.16+0.64+0.09)/5=1.29/5=0.258\). This score describes the forecasts’ squared probability error on these cases; it does not identify which probability range is miscalibrated.
For comparison, assigning every case the observed prevalence, 0.60, gives a Brier score of \([2(0.60^2)+3(0.40^2)]/5=(0.72+0.48)/5=0.240\). The forecast set’s score of 0.258 is higher in this small example. That comparison is descriptive and does not establish that the forecasts will perform worse in another sample.
Logistic Recalibration
A more structured assessment uses logistic recalibration. For probabilities strictly between zero and one, define \(\operatorname{logit}(q)=\log(q/(1-q))\). Fit the model \(\operatorname{logit}\{P(Y_i=1)\}=a+b\operatorname{logit}(q_i)\). Perfect calibration corresponds to intercept \(a=0\) and slope \(b=1\). An intercept different from zero indicates systematic over- or underprediction after accounting for the slope; a slope below one often indicates predictions that are too extreme, while a slope above one often indicates predictions that are not spread out enough. These interpretations should be made with uncertainty intervals and the prediction range in view.
The calibration-in-the-large intercept is commonly estimated with the prediction slope fixed at one, as an offset model. Its fitted value is not generally the difference between the logits of the overall observed and predicted rates. The reason follows directly from the likelihood. With \(p_i(a)=\operatorname{logistic}(\operatorname{logit}(q_i)+a)\), the Bernoulli log-likelihood is \(\ell(a)=\sum_i[Y_i\log p_i(a)+(1-Y_i)\log(1-p_i(a))]\). Since \(dp_i(a)/da=p_i(a)\{1-p_i(a)\}\), differentiation gives \(d\ell/da=\sum_i\{Y_i-p_i(a)\}\). At the maximum, the score is zero, so the fitted intercept solves \(\sum_i p_i(a)=\sum_iY_i\). With varying \(q_i\), this equation depends on the individual predictions.
Worked Example: Fitting an Offset Intercept With Varying Predictions
Consider 100 hypothetical cases: 50 have prediction 0.40, 50 have prediction 0.80, and 50 cases in total have the event. The mean prediction is \((50(0.40)+50(0.80))/100=0.60\), while the observed rate is \(50/100=0.50\). A difference of overall logits would be \(\operatorname{logit}(0.50)-\operatorname{logit}(0.60)=0-\log(1.5)=-0.4055\). That quantity describes the logits of the two overall rates; it is not the fitted offset intercept for these varying predictions.
The offset intercept \(a\) must solve \[ 50\operatorname{logistic}(\operatorname{logit}(0.40)+a)+ 50\operatorname{logistic}(\operatorname{logit}(0.80)+a)=50. \] Dividing by 50, the mean of the two adjusted probabilities must equal 0.50. Numerically solving gives \(a\approx-0.490\). For example, at \(a=-0.490\), the adjusted probabilities are approximately \(\operatorname{logistic}(-0.4055-0.490)=0.290\) and \(\operatorname{logistic}(1.3863-0.490)=0.710\); their mean is approximately 0.500. The offset fit therefore matches the total of 50 events. Individual predictions are needed for this calculation; the two overall rates alone are insufficient.
If all predictions are identical, the simpler logit difference does equal the fixed-slope intercept. For example, if every case has \(q_i=0.60\) and 54 of 100 cases have the event, the score equation becomes \(100\operatorname{logistic}(\operatorname{logit}(0.60)+a)=54\). Hence \(a=\operatorname{logit}(0.54)-\operatorname{logit}(0.60)\approx0.1603-0.4055=-0.2451\). The equality holds here because all predictions have the same value, not merely because their mean is 0.60.
Diagnostics, Comparisons, and Limitations
Use plots and measures together. A nearly zero overall difference can coexist with underprediction at low probabilities and overprediction at high probabilities. A binned plot can reveal that pattern, while a fitted intercept and slope summarize particular aspects of it. Conversely, a fitted slope can be unstable when predictions occupy a narrow range, and a plot can be noisy when bins contain few events. Report uncertainty for calibration estimates where feasible, and avoid treating a jagged curve from sparse data as a reliable pattern.
- Check the event and denominator. Confirm that the observed outcome corresponds to the predicted event, dose, and time horizon, and that missing outcomes have not been treated as non-events.
- Check the data structure. If the same patient contributes several dose scenarios or repeated outcomes, the rows may be dependent. Use methods that account for clustering or summarize at a clearly defined independent unit.
- Check sensitivity to binning. State the number and construction of bins. Compare reasonable alternatives rather than selecting boundaries solely because they make the plot look favorable.
- Check the probability range. A calibration slope or curve may be poorly determined if there are few events or few predictions near the extremes. Exact predictions of zero or one also require care in logit recalibration.
Simulation-based calibration has a distinct interpretation from validation against observed outcomes. If simulated outcomes are generated from the same model and probability definition that produced \(q_i\), agreement is a useful check of implementation and simulation behavior. It does not independently establish that the model represents real patients or that its predictions transport to a new population. When comparing candidate dose rules, calibration should be assessed on predictions and outcomes generated for the same target setting; apparent performance can otherwise reflect a change in population composition rather than a change in the prediction method.
Calibration is also not a dose-selection rule. A well-calibrated toxicity probability does not establish that a dose has acceptable benefit-risk balance, and a calibrated efficacy forecast does not establish that efficacy is clinically meaningful. Calibration evaluates probability agreement, not discrimination, causal effect, or decision utility. The main alternatives answer different questions: a calibration plot emphasizes where probabilities agree or diverge; the overall observed-minus-predicted rate summarizes aggregate bias; logistic recalibration summarizes intercept and slope; and the Brier score summarizes squared forecast error while mixing calibration with other predictive properties. None is a complete substitute for the others.
Common Pitfall
Correction: That difference compares the logits of the two aggregate rates. With varying predictions, the fitted offset intercept solves \(\sum_i\operatorname{logistic}(\operatorname{logit}(q_i)+a)=\sum_iY_i\), which requires the individual \(q_i\). The logit difference is the fitted intercept when all predictions are equal to the mean prediction.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A bin contains 40 cases with mean predicted event probability 0.25 and 12 observed events. What are the observed proportion and the point’s deviation from the calibration diagonal?
- A validation set has 80 events among 200 cases and a mean predicted probability of 0.35. Calculate the overall observed-minus-predicted difference and interpret its sign.
- Why can a near-zero overall difference fail to show that predictions are calibrated across the probability range?
- What does a calibration assessment based only on outcomes simulated from the same model primarily check, and what does it not establish?
Solutions. (1) The observed proportion is \(12/40=0.30\). It is 0.05 above the mean prediction, so the point is \((0.25,0.30)\), above the diagonal. (2) The observed rate is \(80/200=0.40\), and the difference is \(0.40-0.35=0.05\), or five percentage points. Events occurred more often overall than predicted. (3) Overprediction in one range can cancel underprediction in another; inspect a calibration plot or other range-specific assessment. (4) It primarily checks consistency of the prediction and simulation implementation under the assumed model. It does not independently establish calibration against real outcomes or transportability to a target population.