Calibration of Predicted Toxicity and Target Attainment

Why Calibration Matters

A dosing model may assign a patient a 20% probability of toxicity or a 75% probability of attaining a target exposure. These numbers are useful only if they correspond reasonably to event frequencies among comparable patients. Calibration evaluates that correspondence: when patients receive predictions near a stated probability, does the outcome occur at approximately that frequency?

Calibration should be assessed separately for each predicted outcome. For toxicity, define the event and its assessment window in advance, and let \(Y_i^T=1\) if patient \(i\) experiences that event. For target attainment, let \(Y_i^A=1\) if the prespecified target is attained under the specified dosing and measurement conditions. The corresponding predictions are \(\hat p_i^T\) and \(\hat p_i^A\). A prediction is calibrated when, among patients assigned probability \(p\), the event occurs with frequency \(p\), subject to sampling variation.

This is a property of predictions in a defined validation population, for a defined outcome and time window. It does not establish that a model is clinically beneficial, nor does it show that a recommended dose caused an outcome. If predicted outcomes depend on a dosing policy, validation must evaluate outcomes under a sufficiently comparable policy and state that policy clearly.

Calibration-in-the-Large

A common estimate of calibration-in-the-large comes from logistic regression with the original prediction’s log-odds entered as an offset. Write \(\operatorname{logit}(p)=\log\{p/(1-p)\}\) and \(\operatorname{expit}(z)=1/(1+\exp(-z))\). For one binary outcome, fit

where the coefficient of \(\operatorname{logit}(\hat p_i)\) is fixed at 1. The ideal value is \(a_{\mathrm{CITL}}=0\). A positive estimate indicates that the observed event odds are higher than predicted overall; a negative estimate indicates that they are lower. Because this is an odds-scale adjustment, \(\exp(a_{\mathrm{CITL}})\) is the multiplicative adjustment to the predicted odds in this model.

This assessment requires predictions made without using the outcomes being evaluated. It also requires an outcome measured consistently for each patient and a validation sample relevant to the population and dosing conditions where the predictions will be used. The offset model estimates one overall adjustment; it does not by itself establish good calibration at every probability level.

Calibration Slope

The calibration slope is estimated by fitting a logistic model with both an intercept and a coefficient on the prediction’s log-odds:

Ideal calibration has \(a=0\) and \(b=1\). The slope describes how strongly observed log-odds change as predicted log-odds change. A slope below 1 commonly indicates predictions that are too extreme: lower predictions are too low and higher predictions too high, after accounting for the fitted intercept. A slope above 1 commonly indicates predictions that are too compressed. Interpret the intercept from this joint model with care: it is not generally the same as calibration-in-the-large, which fixes the slope at 1.

The score equation explains why calibration-in-the-large adjusts the model toward the observed event count. Let \(q_i=\operatorname{expit}\{a_{\mathrm{CITL}}+\operatorname{logit}(\hat p_i)\}\). For independent Bernoulli outcomes, the log-likelihood, ignoring constants, is

Since \(dq_i/da_{\mathrm{CITL}}=q_i(1-q_i)\), differentiating gives

At an interior maximum, this derivative is zero, so \(\sum_iY_i=\sum_iq_i\). The fitted overall expected event count therefore matches the observed event count. This equality does not require the probabilities to be well calibrated individually: substantial errors at low and high predicted risks can cancel in the total.

Reliability Plots

A reliability plot provides a visual check over the range of predictions. A simple construction groups patients into bins based on predicted probability. For each bin, plot the mean predicted probability on the horizontal axis and the observed event proportion on the vertical axis. The reference line is \(y=x\). Points above it indicate observed frequencies higher than mean predictions in those bins; points below it indicate lower frequencies.

The following hypothetical table concerns target attainment. The displayed differences help show where aggregate predictions and observed frequencies disagree; they do not imply that every patient in a bin has the same risk.

Assumptions, Diagnostics, and Alternatives

These assessments are meaningful only when the prediction and outcome refer to the same event definition, time horizon, and dosing context. For instance, a probability of attaining a concentration target at a specified sampling time cannot be checked against attainment at a different time. A predicted toxicity risk under one dosing policy cannot be fairly evaluated against outcomes generated under a substantially different policy without addressing that difference. Predictions should be generated before the outcomes are observed, and outcome ascertainment should not depend in a way that distorts comparisons on the predicted risk.

Inspect calibration estimates and plots for sparse regions, very few events or nonevents, and predictions near 0 or 1. Such data can lead to unstable estimates or separation in logistic regression. If predictions are exactly 0 or 1, their logits are infinite; any numerical truncation to a small interval such as \([\epsilon,1-\epsilon]\) must be prespecified and reported, since it changes the analysis. For repeated observations per patient or clustered data, uncertainty calculations should account for that dependence, for example by resampling at the patient level rather than treating all records as independent.

Reliability plots depend on how bins are chosen. Equal-width bins can be nearly empty when predictions cluster in a narrow range; equal-count bins can span a wide range and conceal local patterns. Report the binning method and bin sizes. A smooth calibration curve can reveal more detail, but its shape depends on the smoothing method and can be unreliable where data are sparse. Neither a plotted curve nor a single calibration coefficient is a substitute for examining uncertainty and the distribution of predictions.

Calibration differs from discrimination, which concerns how well predictions rank patients with and without the event, and from threshold-based classification summaries. A model can discriminate well yet systematically overpredict risk, or have reasonable overall calibration while ranking patients poorly. Calibration plots and intercepts/slopes address probability agreement; they do not by themselves determine whether using a prediction improves a dosing decision. Evaluation of clinical utility is a separate question.

For both toxicity and target attainment, report the outcome definition, population, prediction range, and validation conditions. Present calibration-in-the-large and the joint-model slope with uncertainty, and show a reliability plot that makes sparse regions visible. These complementary summaries make it easier to detect whether predicted probabilities are systematically shifted, too extreme or compressed, or locally inaccurate.