Calibration and Residual Checks for Binary PK/PD Models

Why Check Calibration?

A binary PK/PD model can rank patients or occasions from lower to higher risk and still give inaccurate probabilities. For example, among observations assigned a 30% probability of a pharmacodynamic event, calibration asks whether the event occurs about 30% of the time. This matters when predictions inform decisions that depend on absolute risk, such as whether an exposure level is associated with an acceptable event probability.

Let \(Y_i\) be the binary response for observation \(i\), with \(Y_i=1\) denoting the event and \(Y_i=0\) its absence. Let \(\hat p_i\) be the model-predicted probability of the event for that observation. A fitted binary model commonly gives \(\hat p_i=\operatorname{logit}^{-1}(\hat\eta_i)\), where \(\hat\eta_i\) is its fitted linear predictor and \(\operatorname{logit}^{-1}(u)=1/(1+\exp(-u))\). Diagnostics compare these probabilities with the outcomes, not merely with the model’s fitted coefficients.

A finite sample will not match predictions exactly. Calibration assessment therefore looks for differences that are large, persistent, or patterned enough to be inconsistent with ordinary outcome variation. The target must also be clear: predictions for a new patient, predictions conditional on a patient-specific random effect, and marginal predictions averaged over patient variability answer different questions. Compare observed outcomes with predictions defined for the same target.

Grouped Observed-versus-Predicted Risks

A direct diagnostic is to sort observations by \(\hat p_i\), divide them into groups, and compare each group’s mean predicted risk with its observed event proportion. If group \(g\) contains \(n_g\) observations and \(O_g\) events, its observed risk is \(O_g/n_g\), while its mean predicted risk is the average \(\hat p_i\) in that group. A calibration plot places these two quantities on the vertical and horizontal axes, respectively. Perfect empirical agreement lies on the 45-degree line.

The grouping is a summary, not a model assumption. Groups should have enough observations and events to make observed proportions informative. Quantile-based groups often keep group sizes similar, while exposure-based groups can answer whether misfit changes over the concentration range. The choice should be reported: different cut points can hide or expose different patterns. Plot uncertainty intervals for observed risks when possible; an observed point away from the line is not automatically evidence of systematic misfit.

The standard error calculation is an approximation when predictions vary within a group. With independent outcomes and fixed predicted probabilities, the variance of the total number of events is \(\sum_i \hat p_i(1-\hat p_i)\); dividing by \(n_g^2\) gives the variance of the observed proportion. This expression is preferable when within-group predictions are not nearly equal. If observations are repeated within patients, treating them as independent can understate uncertainty; use an approach that respects the dependence and the prediction target.

Residuals for Binary Outcomes

For an individual Bernoulli observation, the raw residual is \(Y_i-\hat p_i\). It equals \(1-\hat p_i\) when the event occurs and \(-\hat p_i\) when it does not. Its expected value is zero if the predicted probability is correct, and its variance is \(\hat p_i(1-\hat p_i)\) under the Bernoulli model. Dividing by the square root of that variance gives the Pearson residual.

This standardization follows directly from the Bernoulli variance. If \(Y_i\) has event probability \(p_i\), then \(E(Y_i)=p_i\) and \(\operatorname{Var}(Y_i)=p_i(1-p_i)\). Replacing \(p_i\) with the fitted \(\hat p_i\) gives a residual with approximate unit scale when the model is adequate. It does not make the residual normally distributed: a single binary outcome has only two possible residual values at a given \(\hat p_i\).

Deviance residuals provide another signed measure of each observation’s contribution to the Bernoulli log-likelihood discrepancy from a saturated fit. For \(Y_i\) equal to zero or one, they are

Terms of the form \(0\log(0/q)\) are defined as zero by continuity. For the four observations above, the deviance residuals are approximately \(-0.668,\ 1.354,\ 0.845,\) and \(-1.794\). For example, the first is \(-\sqrt{2\log(1/0.80)}=-0.668\), and the second is \(+\sqrt{2\log(1/0.40)}=1.354\). Pearson and deviance residuals are useful for finding unusual observations, but neither should be expected to follow a normal distribution in a binary model.

Look for Patterns Across Exposure

A residual’s sign has a direct interpretation: positive values indicate an event occurred more strongly than predicted, while negative values indicate less response than predicted. Plot residuals or grouped observed-versus-predicted risks against exposure, predicted risk, time, dose, and other scientifically relevant covariates. A trend can point to a missing or misspecified exposure effect, a changing baseline risk, or another omitted structure. A random-looking spread is reassuring but cannot prove that the model is correct.

For a calibration-in-the-large check, one may fit a logistic recalibration model with the original predictions as an offset:

The ideal intercept is \(a=0\); a positive estimate indicates that observed risks are generally higher than predicted, and a negative estimate indicates they are generally lower. A calibration-slope check instead fits \(\operatorname{logit}\{P(Y_i=1)\}=a+b\operatorname{logit}(\hat p_i)\); ideal values are \(a=0\) and \(b=1\). A slope below one often indicates that predictions are too extreme, while a slope above one indicates insufficient spread, provided the recalibration model is appropriate. These summaries can conceal local deviations, so they complement rather than replace plots and exposure-stratified checks.

Assumptions, Alternatives, and Failure Modes

The diagnostics rely on the outcome and prediction being correctly paired, the event definition being consistent, and the probabilities referring to the same population and prediction target as the observed data. Simple binomial uncertainty calculations further assume independent Bernoulli outcomes with the stated probabilities. For repeated observations, clustered data, or predictions estimated from the same data used to assess them, uncertainty needs to reflect dependence and model fitting. An external validation sample or resampling that repeats the full fitting process can help assess performance beyond the development data.

Calibration plots and residuals answer different questions from other model checks. A discrimination measure summarizes ranking ability, not whether a predicted 0.30 risk occurs about 30% of the time. A Brier score, the mean squared difference between \(Y_i\) and \(\hat p_i\), summarizes probabilistic prediction error but combines calibration and discrimination and does not reveal where misfit occurs. Likelihood-based comparisons assess relative fit among specified models; they do not by themselves show that absolute risks are calibrated. No single metric replaces inspection of the predictions and their context.

A grouped goodness-of-fit test can reduce a complex diagnostic to one statistic, but its result depends on the grouping and sample size. A non-significant result does not demonstrate calibration, and a small departure can be statistically detectable in a large sample without being clinically important. Conversely, sparse events can make plots and formal tests uninformative. Avoid treating arbitrary bins, a single influential residual, or a single p-value as a complete assessment.

• Pitfall: treating a point off the diagonal as proof of miscalibration. Observed risks vary by chance. Compare the discrepancy with its uncertainty and look for coherent patterns.
• Pitfall: declaring the model adequate because a grouped test is not significant. The test may have low power, and grouping can hide local errors. Examine the curve, residual patterns, and clinically relevant exposure strata.
• Pitfall: checking only overall calibration. Average overprediction at low exposure and underprediction at high exposure can cancel. Stratify or plot against exposure and other important covariates.
• Pitfall: using in-sample agreement as evidence of predictive performance. Fitted predictions can look optimistic. Assess predictions on data not used for fitting or use an appropriate resampling procedure.

The practical goal is not to make every observed proportion equal its prediction. It is to identify whether discrepancies are larger or more systematic than expected, whether they occur where decisions depend on accurate risk, and whether the model’s prediction target and data structure justify the uncertainty calculation.