Representing Model Discrepancy in Decision Predictions

Why Model Discrepancy Matters for Decisions

A pharmacometric model can be estimated precisely and still make systematically inaccurate predictions. Parameter uncertainty describes uncertainty about the values of parameters within the chosen model. Model discrepancy is systematic lack of fit that remains because the model’s predictions do not fully represent the data-generating process. If a decision prediction is based only on parameter uncertainty, this remaining mismatch can make its apparent precision misleading.

For example, a model might capture average response across the observed dose range but consistently underpredict response at higher doses. A narrow interval for its parameters does not make that high-dose prediction reliable. Residual patterns can provide evidence of this problem; an explicit discrepancy term can represent a correction and uncertainty about that correction. Neither step proves that the model is wrong in a particular way. Data issues, a poorly specified observation model, or an unrepresented feature of the study can also produce residual patterns.

The terms describe different sources of uncertainty. Parameter uncertainty concerns \(\theta\); discrepancy concerns systematic differences between \(f\) and the process of interest; residual variability concerns deviations of individual observations around the discrepancy-adjusted mean. A future-observation prediction may include all three. A prediction of a latent mean response generally excludes the future observation’s residual variability. The decision target therefore determines which uncertainty components belong in the prediction.

Discrepancy and parameter uncertainty may be difficult to distinguish: a parameter change can sometimes compensate for a systematic mismatch. They are not automatically identifiable from the data. Their separation depends on the model, the data’s coverage of relevant conditions, and any defensible assumptions used to constrain the discrepancy. Residual patterns are evidence to investigate, not a unique diagnosis.

From Residual Patterns to a Decision Prediction

Residuals compare observations with fitted predictions. If residuals show a trend with dose, time, exposure, or another relevant input, the model may be systematically missing a relationship. A plot can make such structure visible: residuals should not display an obvious pattern if the model adequately captures the mean relationship, though random residuals alone do not establish that the model is correct.

Before treating a pattern as discrepancy, check the data, units, study design, and observation model. Consider whether the pattern could reflect an influential observation, a protocol difference, or an inappropriate residual structure. Then ask whether the pattern occurs in a region relevant to the decision. A mismatch far from the decision’s target conditions may have little impact; a small bias near a go/no-go threshold may change the decision materially.

For a decision-relevant prediction \(Y^*\), a general representation is \(Y^* = f(x^*;\theta)+\delta(x^*)+\epsilon^*\). When the components are independent and the residual has mean zero, the predictive mean and variance are

The variance expression is not a license to assume independence. In general, the variance also includes twice each pairwise covariance, such as \(2\operatorname{Cov}\{f(x^*;\theta),\delta(x^*)\}\). If discrepancy is correlated across inputs, its covariance structure matters when predicting several outcomes or a function of outcomes. For a decision about a latent mean, omit \(\epsilon^*\); for a decision about an individual future observation, include it if it is part of the target.

A useful justification follows from the variance decomposition. For independent components, subtract the predictive mean from \(Y^*\), square, and take expectations. The cross-products have expectation zero because each centered component is independent of the others and has mean zero. The remaining expected squares are the component variances, giving the expression above. Thus adding discrepancy uncertainty can widen a prediction even when the discrepancy’s expected value is zero; a nonzero expected discrepancy can also shift the prediction itself.

Representing Discrepancy Without Overstating What Is Known

A discrepancy term should match the evidence and the decision target. It may be a constant offset over a relevant range, a smooth function of an input, or a structured random function whose values at nearby inputs are related. The choice should be explicit: it determines how observed residual patterns affect predictions at unobserved conditions. A correction estimated at one dose does not automatically justify extrapolating a slope across all doses.

For an uncertain discrepancy, state its expected correction and uncertainty at the decision-relevant inputs. If the discrepancy is represented as a random function, its covariance describes how uncertainty at one input relates to uncertainty at another. If it is a fixed correction, explain how the correction was selected and what uncertainty remains around it. Avoid silently treating a discrepancy estimate as known exactly.

Parameter and discrepancy uncertainty can also be dependent. For example, a parameter value that fits the observed data well may be associated with a larger discrepancy correction. In that case, propagating each uncertainty separately and then adding variances as if independent is not justified. The joint dependence must be preserved or the independence assumption stated as a simplifying approximation.

Diagnostics, Alternatives, and Limitations

Useful diagnostics include residual plots against important inputs, time, fitted values, and study or occasion; checks for persistent runs of residuals with the same sign; and comparison of predictions with held-out or external data when available. Inspect whether a pattern is concentrated in a particular study or measurement range. Check the influence of individual observations and whether the pattern is large enough to affect the decision. If repeated measurements or related outcomes are involved, examine whether residual dependence has been represented appropriately.

These checks are imperfect. A residual plot may fail to reveal discrepancy when data are sparse, noisy, or poorly distributed over the decision-relevant region. Conversely, a visible pattern may reflect data quality or an inadequate residual model rather than a missing systematic correction. A discrepancy function can be weakly identified, especially outside the observed input range. Its uncertainty should generally increase when a decision requires extrapolation unsupported by data.

The earlier material on structural uncertainty addresses uncertainty among candidate model structures. Discrepancy is related but distinct: it represents lack of fit not captured by the selected model, and may remain even if one structure is chosen. In practice, structural uncertainty and discrepancy can coexist. This tutorial’s calculation should not be read as a substitute for comparing scientifically credible model structures or investigating data and design.

A common mistake is to say, “The parameter estimates are precise, so the decision prediction must be precise.” This is wrong because precision conditional on a model does not account for systematic model mismatch. The correction is to assess residual patterns and represent relevant discrepancy and its uncertainty when forming the decision prediction. The opposite mistake is to add a large arbitrary error term “to be conservative.” That may obscure evidence and distort decisions; discrepancy assumptions need a rationale, and their influence should be reported transparently.