Why Diagnose an Interim Model Before Using It?
An interim pharmacometric model can produce a precise-looking estimate and still be a poor basis for adaptation. If its concentration predictions are systematically high at early times, or its parameters cannot be distinguished with the available sampling, then a dose decision based on predicted exposure may be unstable. Diagnostics ask whether the model reproduces relevant features of the observed data and whether the data actually inform the parameters needed for the decision.
No single diagnostic proves a model correct. Residual checks examine discrepancies between observations and predictions; visual predictive checks (VPCs) compare observed summaries with summaries from replicated data generated under the fitted model; identifiability diagnostics assess whether different parameter values can be distinguished by the available design and data. These forms of evidence answer related but different questions.
Residual Checks: Look for Structure, Not Just Large Values
Suppose a concentration measurement \(C_{ij}\) for participant \(i\) at observation time \(t_{ij}\) follows a lognormal observation model:
Here \(\widehat{C}_{ij}\) is the model prediction appropriate to the diagnostic being used, and \(\sigma\) is the log-scale residual standard deviation. This model assumes positive concentrations, multiplicative measurement variability, and normally distributed log errors with constant variance. If variability changes with concentration, observations are correlated, or the residual distribution is not approximately normal, this simple standardization may not be appropriate.
A log-scale standardized residual is
Under the stated observation model with known parameters and an appropriate prediction, residuals should be centered near zero and should not show systematic patterns against time, predicted concentration, dose, or covariates. In practice, parameters are estimated and observations may be used in fitting the model, so residuals need not be independent standard normal values. Conditional predictions can also look reassuring because they use information from the same participant's data. Predictive or simulation-based residuals, and comparisons with population-level predictions, can help assess this concern.
Worked Example: Reading Log-Scale Residuals
Hypothetical data. At three sampling times, observed concentrations are 12.2, 7.3, and 4.1 mg/L. Corresponding model predictions are 10.0, 6.0, and 4.0 mg/L. Assume the lognormal observation model above with \(\sigma=0.20\), and use the stated predictions for this calculation.
For the first observation, \(r=\log(12.2/10.0)/0.20=\log(1.22)/0.20\approx0.19885/0.20\approx0.994\). For the second, \(r=\log(7.3/6.0)/0.20\approx0.1961149/0.20\approx0.981\). For the third, \(r=\log(4.1/4.0)/0.20=\log(1.025)/0.20\approx0.02469/0.20\approx0.123\). Calculations retain precision until the final rounding.
All three residuals are positive, so each observation exceeds its prediction. A few observations are not enough to establish a systematic bias, but if positive residuals repeatedly occur early after dosing, the model may underpredict early concentrations. That pattern matters if the adaptation depends on peak exposure. A residual near one is not, by itself, a universal rejection threshold; interpretation depends on the residual construction, sample size, repeated-measure structure, and prespecified diagnostic plan.
Useful displays include residuals against time after dose, predicted concentration, dose, and important covariates, as well as participant-level or sequence plots where repeated observations could conceal within-person patterns. A funnel shape can suggest concentration-dependent variance; a curved time trend can suggest a structural discrepancy; clusters by dose or covariate can indicate a missing or inadequate relationship. Inspect influential observations and data-processing issues as well: a single unit conversion or timing error can create an apparent model failure.
Visual Predictive Checks: Does the Model Reproduce the Data?
A VPC evaluates the model by simulation. Fit the model, simulate many replicated data sets under the fitted parameters, and reproduce the trial features that affect the data: dose, covariates, sample size, sampling times, and observation process. In each time or design stratum, compare observed summaries, such as medians and upper percentiles, with the corresponding simulated summaries. A simulation envelope for a summary shows the range of values the fitted model commonly produces under repetition; it is not a confidence interval for a model parameter.
The comparison must respect the data structure. Pooling observations across doses or sampling times can hide a discrepancy in a decision-relevant stratum. Sparse or uneven sampling makes empirical percentiles noisy, so apparent departures should be considered alongside sample size and simulation variability. Censoring, missingness, and transformations should be handled consistently with the analysis model and observed data.
Worked Example: Interpreting a VPC Envelope
Hypothetical data. At an early post-dose time, 40 participants contribute concentrations. The observed median is 10.2 mg/L and the observed 90th percentile is 15.1 mg/L. Across replicated data sets simulated from the fitted model, the central 90% envelope for the median is 9.2 to 10.8 mg/L, and the corresponding envelope for the 90th percentile is 12.8 to 14.6 mg/L.
The median, 10.2 mg/L, falls within its simulation envelope. The 90th percentile, 15.1 mg/L, exceeds the upper envelope boundary of 14.6 mg/L by \(15.1-14.6=0.5\) mg/L. Thus the model reproduces the center of this stratum reasonably in this comparison but may underrepresent its upper tail. This is a diagnostic signal, not proof of a particular cause: possible explanations include an inadequate variability model, unmodeled heterogeneity, or a small-sample fluctuation.
If adaptation depends on the proportion of participants above an exposure threshold, the upper-tail discrepancy deserves targeted investigation even though the median matches. The next checks should include the relevant dose and covariate strata, data quality, and whether simulations reproduced the actual sampling and observation process.
Identifiability: Can the Data Distinguish the Parameters?
A parameter is structurally identifiable if, in principle, distinct values produce distinct observable-data distributions under the model and design. It is practically identifiable if the available data estimate it with useful precision. A structurally identifiable parameter can still be practically weakly identified when the sample is small, observations are noisy, or sampling times provide little information. Conversely, a numerically converged optimizer does not establish identifiability.
A local diagnostic uses model sensitivities. Let \(m_j(\theta)\) be the model prediction for observation \(j\), with parameter vector \(\theta\). The sensitivity matrix has entries \(J_{jk}=\partial m_j(\theta)/\partial\theta_k\). If columns are nearly linearly dependent, different parameter changes produce similar changes in predictions. Under an approximately normal observation model with independent errors of common variance \(\sigma^2\), the local information matrix is proportional to \(J^\mathsf{T}J/\sigma^2\). A rank-deficient matrix signals a local direction that the data do not inform; a poorly conditioned matrix signals a weakly informed combination. Scaling parameters before interpreting condition numbers is important because parameter units affect numerical conditioning.
Worked Example: Sampling Times and Local Identifiability
Hypothetical model. Suppose log concentration follows \(\log C(t)=a-kt+\epsilon\), where \(a=\log A\), \(k\) is an elimination rate in inverse hours, and independent log errors have known standard deviation \(\sigma=0.20\). For the parameter vector \((a,k)\), the sensitivity row at time \(t\) is \((1,-t)\).
With observations at 1.0 and 1.1 hours, the sensitivity matrix is \(J=\begin{pmatrix}1&-1.0\\1&-1.1\end{pmatrix}\). Therefore, \(J^\mathsf{T}J=\begin{pmatrix}2&-2.1\\-2.1&2.21\end{pmatrix}\), whose determinant is \(2(2.21)-(2.1)^2=4.42-4.41=0.0100\). The small determinant indicates that these two closely spaced observations provide little separation of \(a\) and \(k\).
If the times are instead 1.0 and 3.0 hours, then \(J^\mathsf{T}J=\begin{pmatrix}2&-4\\-4&10\end{pmatrix}\) and its determinant is \(20-16=4\), much larger. The wider time separation provides more information about the slope \(k\) relative to the intercept \(a\), under this model. Multiplying either information matrix by \(1/\sigma^2=25\) changes its scale but not its rank or relative conditioning. This local calculation assumes the model and error structure are suitable; it does not account for all sources of uncertainty or establish global identifiability.
Other practical warning signs include large standard errors, strong parameter correlations, flat or elongated likelihood or posterior regions, estimates near imposed boundaries, and substantial changes under small data or initialization perturbations. Examine uncertainty in the actual decision-relevant prediction as well as in individual parameters: parameters may be correlated yet yield a stable prediction at one target, or apparently precise estimates may combine into an unstable prediction elsewhere. Profile likelihoods, posterior plots, and simulation-based parameter recovery can provide complementary evidence.
Combining Diagnostics for an Adaptation Decision
Residual plots, VPCs, and identifiability checks are complementary. Residual plots can locate patterns at the observation level; a VPC asks whether replicated data resemble observed distributions; identifiability analysis asks whether the available design supports the parameters and predictions being used. A model can pass one check and fail another. For example, a good median VPC does not rule out weak identification of a clearance parameter, and a well-identified parameter does not guarantee that the model describes the data adequately.
Before using diagnostics, define which predictions matter to the adaptation, such as exposure at a specified dose, a concentration threshold probability, or a covariate-specific prediction. Then examine those quantities at the relevant times, doses, and subgroups. Prespecify diagnostic displays and criteria where feasible, and document departures, model revisions, and sensitivity of the proposed decision. There is no universal residual cutoff or VPC pass/fail rule that substitutes for this context.
If evidence of inadequacy is decision-relevant, possible responses include checking data and implementation, collecting or prioritizing informative samples, revising a defensible model component, or delaying the adaptation pending further information. These actions should follow the trial's prespecified governance and decision algorithm. Diagnostics cannot guarantee that a model is true, and unplanned model changes made solely to obtain a favorable decision can undermine the credibility of the adaptation.
Practical Diagnostic Sequence
Name the adaptation-relevant prediction, dose, time, and population before interpreting fit.
Inspect residuals against time, predictions, dose, and covariates; account for repeated observations and prediction type.
Simulate the trial's dose, covariate, sampling, and observation features, then compare relevant distributions by stratum.
Review sensitivities, parameter uncertainty and correlation, boundary behavior, and uncertainty in target predictions.
Record whether departures could change the adaptation and follow the prespecified response process.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Under a lognormal observation model, what does a positive standardized log residual indicate?
- Why should a VPC reproduce the trial's dose and sampling-time structure?
- Can optimizer convergence establish that a parameter is identifiable? Explain.
- In the sensitivity example, why is a small determinant of \(J^\mathsf{T}J\) concerning?
- If a VPC median agrees with observations but the decision-relevant upper tail does not, what should be examined?
Solutions. (1) The observed concentration exceeds the prediction on the log scale, relative to the assumed residual standard deviation. (2) The simulated summaries must be comparable with the observed summaries; changing dose or sampling structure can mask or create discrepancies. (3) No. Convergence is a numerical property of the fitting procedure, not evidence that distinct parameter values can be distinguished by the data. (4) It indicates that the sensitivity directions are close to linearly dependent or weak overall, so some parameter combination is poorly informed locally. (5) Inspect the relevant stratum, variability and heterogeneity assumptions, data quality, and simulation design; determine whether the tail discrepancy could alter the adaptation rather than relying on the median alone.