Diagnosing External Calibration

What Calibration Adds to External Prediction Assessment

A model can produce predictions that are, on average, close to external observations yet still be poorly calibrated. For example, it may systematically underpredict low concentrations and overpredict high concentrations. A single average error can conceal that pattern. Calibration assessment asks whether predictions agree with observations in level and spread across the prediction range.

Here, \(Y_{ij}\) denotes the observed concentration for subject \(i\) at record \(j\), and \(\hat{Y}_{ij}\) denotes the corresponding external prediction, generated according to the prespecified prediction target. Calibration concerns the agreement between these paired quantities in the external validation sample. The prediction target, outcome scale, and analysis set must be fixed before examining the validation results.

The chosen scale matters. On the concentration scale, differences and offsets are in concentration units, such as mg/L. On a log-concentration scale, differences represent log ratios: a positive mean difference indicates that observed concentrations tend to exceed predictions multiplicatively. Use a log scale only when observations and predictions are positive and that scale is appropriate to the scientific question. Do not mix scales within a calibration analysis or add arbitrary constants to make zero values log-transformable.

Calibration-in-the-Large

For \(N\) paired records, define the residual as observed minus predicted. Calibration-in-the-large is the mean of these residuals. It answers a focused question: if the predictions were shifted by one constant amount, what average shift would align them with the observations in this validation sample? It does not show whether the error changes across the prediction range.

This mean is also the fitted intercept in a regression with the coefficient of prediction fixed at 1: \(Y_{ij}=\Delta+\hat{Y}_{ij}+\epsilon_{ij}\). It is not generally the same as the intercept from an unconstrained regression, because the unconstrained regression estimates both intercept and slope. The result depends on the weighting: a record-level mean gives more weight to subjects with more records. If the target is an average subject rather than an average record, define a subject-weighted summary instead.

Calibration Intercept and Slope

Fit the descriptive regression of observed concentration on predicted concentration:

Here \(a\) is the calibration intercept, \(b\) is the calibration slope, and \(\epsilon_{ij}\) is the regression residual. Perfect calibration corresponds to \(a=0\) and \(b=1\). The intercept represents the fitted observed value when the prediction is zero, so it may have little practical meaning if zero lies far outside the prediction range. The slope describes how the observed values change across the prediction range relative to the predicted values.

A slope below 1 means that observed values vary less across the range than the predictions do: predictions are too spread out, or over-dispersed, relative to observations. A slope above 1 indicates predictions are too compressed, or under-dispersed. These interpretations are conditional on the fitted linear relationship and should be checked against a plot. An intercept can offset the line without correcting an incorrect slope.

The ordinary least-squares slope is the centered cross-product of observations and predictions divided by the centered sum of squares of predictions. This expression also shows why the slope describes relative spread.

To see the spread interpretation, suppose the observations follow \(Y=a+b\hat{Y}+\epsilon\), where the regression residual has mean zero and is uncorrelated with the predictions. Then the covariance of prediction and observation is \(b\) times the variance of prediction. Dividing that covariance by the prediction variance gives slope \(b\). Thus a slope below 1 indicates a smaller observed range per unit of predicted range, under these conditions.

Reading the Observed-Versus-Predicted Plot

Plot observed values on the vertical axis and predictions on the horizontal axis, using the same scale and units on both axes. Add the identity line \(Y=\hat{Y}\). Points consistently above the line indicate underprediction; points below it indicate overprediction. A fitted calibration line helps summarize overall level and slope, but the point cloud can show departures that a line conceals.

A calibration plot is a diagnostic summary, not a substitute for checking the data and prediction process. Mark or otherwise distinguish repeated records from the same subject when useful, and avoid treating a dense cluster of records from one subject as if it were many independent subjects. If the plot is curved or shows systematic concentration-dependent departures, report that the linear calibration summaries are incomplete rather than claiming that a slope near 1 establishes calibration.

Assumptions, Uncertainty, and Common Failure Modes

The arithmetic definitions of mean residual and fitted regression do not require normally distributed outcomes. However, inferential standard errors and confidence intervals require an appropriate sampling and dependence treatment. NLME validation data often contain repeated observations within subjects; records from one subject are generally not independent. Ordinary regression standard errors that treat all records as independent can therefore be too small.

For inference, resample subjects rather than individual records in a subject-level bootstrap, retaining each selected subject's paired observations and predictions. Alternatively, use a variance estimator appropriate to the clustering and design. The point estimate still needs a clear target: record-weighted calibration gives each record equal weight, while subject-weighted calibration first summarizes within subject and then gives each subject equal weight. These answer different questions when subjects contribute unequal numbers of records.

The validation sample must represent the intended external prediction setting, and observed outcomes must be paired with predictions generated for the same individuals, times, dosing histories, and covariates. Calibration diagnostics do not repair mismatched data or a changed prediction target. In particular, population predictions and individual predictions are distinct targets; using information from an individual's validation outcomes to construct an ostensibly external individual prediction would compromise the assessment.

• Scale mismatch: Comparing a log-scale prediction with an untransformed observation makes the summaries uninterpretable. Transform both consistently and state the scale and units.
• Reading slope without intercept: A slope near 1 can coexist with a large constant offset. Assess calibration-in-the-large and the fitted intercept as well.
• Reading average bias as universal bias: A mean residual near zero can result from positive and negative errors that cancel. Inspect the observed-versus-predicted plot.
• Ignoring clustering: Many records do not necessarily provide the same information as many independent subjects. Account for within-subject dependence when quantifying uncertainty.
• Extrapolating beyond the data: The fitted intercept at prediction zero may be scientifically irrelevant when zero is outside the observed prediction range. Interpret the fitted line within the range represented by validation data.

Calibration summaries complement, rather than replace, predictive scoring. Scores summarize prediction errors according to their chosen loss function; calibration asks whether observed levels and spread align with predicted levels and spread. Neither alone establishes that a model will predict well in every subgroup or setting. A calibration slope is a descriptive relationship, not a causal effect and not a guarantee of future performance.