Prediction Discrepancies and NPDEs

Why the Scale of a Discrepancy Matters

A difference between an observed concentration and a model prediction is easy to calculate, but its size alone does not say whether it is surprising. A discrepancy of \(2\ \text{mg/L}\) may be large when predictive variation is \(0.5\ \text{mg/L}\), yet modest when predictive variation is \(5\ \text{mg/L}\). Repeated measurements add another complication: discrepancies at two occasions may move together, so treating each as an independent piece of evidence can exaggerate how much information the profile contains.

Normalized prediction distribution errors (NPDEs) address these issues by using each observation's predictive distribution and, for a profile, adjusting for predictive correlation. This tutorial contrasts that approach with raw discrepancies. It does not replace inspection of predictions in their original units: raw differences remain useful for understanding the size and direction of a model's errors. The key is to know which question each summary answers.

Definition: For observation \(i\), let \(y_i\) be the observed concentration and let \(m_i\) be a chosen predictive reference, such as the predictive mean or median. The raw prediction discrepancy relative to that reference is \(d_i=y_i-m_i\), in concentration units. If \(F_i\) is the model-based predictive cumulative distribution function for that observation, the prediction distribution error is \(u_i=F_i(y_i)\). Its normal-score transform is \(z_i=\Phi^{-1}(u_i)\), where \(\Phi\) is the standard normal cumulative distribution function. NPDEs additionally adjust the vector of normal scores for its predictive correlation.

These quantities should not be conflated. A raw discrepancy \(d_i\) is a distance from a reference prediction. A prediction distribution error \(u_i\) is a percentile: it records where the observation lies in its predictive distribution. The normal-score \(z_i\) expresses that percentile on a standard normal scale. For example, \(z_i=1.282\) corresponds approximately to the 90th predictive percentile. It does not mean the observation is \(1.282\ \text{mg/L}\) above a prediction.

From Predictive Distributions to Normal Scores

Suppose the fitted model and design generate a predictive distribution for each observation under the model. In practice, this distribution is commonly approximated by simulated predictions that reproduce the relevant dosing, sampling, and observation structure. The marginal transform \(u_i=F_i(y_i)\) accounts for the location, spread, and shape of the predictive distribution at that observation. If the predictive distribution is continuous and correctly specified, the probability integral transform \(F_i(Y_i)\) is uniform on \((0,1)\); applying \(\Phi^{-1}\) then gives a standard normal variable.

This is a marginal statement. It does not imply that the normal scores from one individual's repeated observations are independent. If predictive scores have covariance matrix \(\Sigma\), the NPDE calculation applies a decorrelation transformation. For a nonsingular covariance matrix, one possible choice is a matrix \(W\) satisfying \(W\Sigma W^\mathsf{T}=I\), where \(I\) is the identity matrix. The transformed vector \(Wz\) has identity covariance when \(z\) has covariance \(\Sigma\). Different valid whitening transformations can differ in orientation; a Cholesky transformation in particular depends on the ordering of observations. The same documented convention should be used when comparing diagnostics.

Interpretation: A raw discrepancy retains the concentration scale and shows direction and absolute size relative to its reference. A marginal normal score shows how extreme the observation is relative to its own predictive distribution. A decorrelated NPDE also accounts for predictive dependence across observations. Under a correctly specified predictive model and an adequate computation, NPDEs should be approximately centered at zero with unit variance and little residual correlation. This is a diagnostic reference, not a guarantee that every finite sample will look exactly normal.

Worked Examples

Worked Example: The Same Raw Difference, Different Predictive Scales

Consider two hypothetical concentration observations. Each is \(2.0\ \text{mg/L}\) above its predictive mean. For a simple comparison, suppose each predictive distribution is approximately normal and symmetric, with the stated predictive standard deviation.

For observation A, the predictive mean is \(10.0\ \text{mg/L}\), the observed value is \(12.0\ \text{mg/L}\), and the predictive standard deviation is \(1.0\ \text{mg/L}\). Thus \(d_A=12.0-10.0=2.0\ \text{mg/L}\), and the normal-score equivalent is

z_A=\frac{12.0-10.0}{1.0}=2.00.

For observation B, the predictive mean is \(20.0\ \text{mg/L}\), the observed value is \(22.0\ \text{mg/L}\), and the predictive standard deviation is \(4.0\ \text{mg/L}\). Thus \(d_B=22.0-20.0=2.0\ \text{mg/L}\), but

z_B=\frac{22.0-20.0}{4.0}=0.500.

The raw discrepancies are identical, but the first observation is two predictive standard deviations above its mean while the second is only half a standard deviation above. This division is appropriate here because normality and the predictive standard deviations were stipulated. In a skewed or otherwise non-normal distribution, the predictive CDF is the more general way to scale the observation.

Worked Example: A Skewed Predictive Distribution

Suppose a hypothetical observation is \(8.0\ \text{mg/L}\) above the predictive median. A simulated predictive distribution is right-skewed: only 10% of its predicted values exceed the observed value, so \(F(y)=0.900\). The observed-minus-median discrepancy gives the direction and magnitude in \(\text{mg/L}\), but not its percentile. The normal-score transformation is

z=\Phi^{-1}(0.900)\approx 1.282.

The interpretation is that the observation lies near the 90th percentile of this predictive distribution, corresponding to a positive normal score of about 1.282. It is not valid to divide \(8.0\ \text{mg/L}\) by an arbitrary standard deviation and assume that result has the same interpretation: skewness means equal distances on either side of the median need not correspond to equal tail probabilities. This example is marginal; it has not yet adjusted for correlation with other observations.

Worked Example: Correlation Adjustment for Two Measurements

Consider two hypothetical normal scores from one concentration profile, \(z=(1.50,1.50)^\mathsf{T}\). Suppose simulations under the fitted model estimate the predictive covariance matrix as

\Sigma= \begin{pmatrix} 1 & 0.80\\ 0.80 & 1 \end{pmatrix}.

The scores move together strongly under the predictive model. Using the Cholesky factor \(L\) satisfying \(\Sigma=LL^\mathsf{T}\), one obtains

L= \begin{pmatrix} 1 & 0\\ 0.80 & 0.60 \end{pmatrix}, \qquad L^{-1}z= \begin{pmatrix} 1.50\\ (1.50-0.80(1.50))/0.60 \end{pmatrix} = \begin{pmatrix} 1.50\\ 0.500 \end{pmatrix}.

The second transformed coordinate is the part of the second score not explained by the first under this particular ordered Cholesky transformation. Although both marginal scores equal 1.50, the second adds less independent information because high scores at both occasions are already common together in the predictive model. The transformed values are illustrative of this factorization and ordering; they are not evidence that the observed profile is adequately described by the model.

Why Decorrelation Works

The covariance adjustment is not an extra rescaling of each concentration by its standard deviation. It is a joint transformation that removes linear dependence among the normal scores under the estimated predictive covariance. For the Cholesky choice, write \(z=Lq\), where \(\Sigma=LL^\mathsf{T}\). Then \(q=L^{-1}z\), and if the covariance of \(z\) is \(\Sigma\),

\operatorname{Cov}(q) =L^{-1}\Sigma(L^{-1})^\mathsf{T} =L^{-1}LL^\mathsf{T}(L^{-1})^\mathsf{T} =I.

This establishes the covariance result: under the assumed covariance, the transformed coordinates have unit variances and zero pairwise covariances. It does not by itself establish that they are independent or normally distributed. Those stronger properties require an appropriate joint normal model. NPDE interpretation therefore depends on the predictive model, the simulation-based estimates of its distributions and covariance, and the adequacy of the transformation.

Comparing Diagnostic Summaries

Summary	What it retains or adjusts	Useful question	Main caution
Raw discrepancy \(y_i-m_i\)	Direction and absolute size in concentration units	How far, and in which direction, is the observation from the chosen reference?	Does not account for varying predictive spread or shape
Marginal normal score \(\Phi^{-1}(F_i(y_i))\)	Each observation's predictive percentile and marginal scale	How surprising is this observation relative to its own predictive distribution?	Repeated scores can remain correlated
NPDE	Marginal scaling and an adjustment for predictive correlation	How does the profile compare with a decorrelated predictive reference?	Depends on estimated predictive distributions, covariance, and transformation choices

These summaries are complementary rather than competing measures of the same thing. Raw discrepancies are often clearest for clinical and operational interpretation because they remain in \(\text{mg/L}\). Marginal scores support comparison across observations with different predictive spreads or shapes. NPDEs are useful when the joint profile matters and the covariance adjustment is reliable. None of them identifies the cause of a discrepancy: a pattern may reflect model misspecification, data problems, or features of the design that make the predictive distribution difficult to estimate.

Assumptions, Diagnostics, and Failure Modes

An NPDE is only as informative as the predictive distribution used to calculate it. The simulated predictions must correspond to the observed data structure and the model being assessed. The marginal CDF must be estimated adequately near the observation; the covariance matrix must be estimated from enough appropriate simulations and be sufficiently well-conditioned to transform. For the usual normal-reference interpretation, the predictive model must also describe the relevant distributional behavior. When these requirements fail, a striking NPDE may reflect a computational or design problem rather than a clear model defect.

Too few predictive simulations. Empirical CDF values are coarse, especially in the tails. A score near an endpoint can become unstable or undefined when transformed by \(\Phi^{-1}\). Use adequate simulation counts and the numerical safeguards established for NPDE computation; examine whether tail conclusions persist with more simulations.
Unreliable covariance estimates. A nearly singular estimated covariance matrix can make decorrelation unstable. Check eigenvalues, conditioning, and sensitivity to simulation count. Do not interpret extreme transformed values without checking the covariance computation.
Residual dependence after adjustment. If transformed scores still show substantial correlation, the predictive covariance estimate or chosen transformation may be inadequate, or the joint predictive structure may not be captured. A unit marginal variance alone does not establish successful decorrelation.
Normality treated as automatic. The probability integral transform gives uniform marginals under a continuous, correct predictive distribution, but estimated CDFs, finite simulations, and discrete or censored observations complicate that ideal. Inspect the simulation and observation handling rather than assuming a perfect normal reference.
Loss of scale and context. A score indicates relative extremeness, not the concentration error in \(\text{mg/L}\). When a score flags an observation, return to the observed and predicted concentrations and their units.

A practical sequence is to inspect raw discrepancies for direction and magnitude, check marginal predictive percentiles to assess scale, and use decorrelated NPDEs when profile dependence is relevant and estimable. Compare the conclusions across these views. A large raw difference that is ordinary under a wide predictive distribution is not equivalent to a large positive NPDE; likewise, several similar marginal scores in a highly correlated profile do not represent several independent deviations.

Common pitfall: “Both observations are \(2\ \text{mg/L}\) above prediction, so they are equally surprising; and two high scores are twice the evidence of one.” The first claim ignores different predictive distributions, and the second ignores dependence. Compare observations using their predictive CDFs, then account for correlation when interpreting the joint profile. Retain raw differences as a separate, unit-bearing description rather than treating them as normalized errors.

Check Your Understanding

Answer each question, then compare with the concise solutions below.

Two observations are each \(1.5\ \text{mg/L}\) above their predictive means. Their predictive standard deviations are \(0.5\ \text{mg/L}\) and \(3.0\ \text{mg/L}\). Under normal predictive distributions, calculate both normal scores and explain the difference.
An observation is at the 2.5th percentile of its predictive distribution. What is its approximate marginal normal score, and what does the sign mean?
Why do marginal normal scores not, by themselves, resolve dependence between two measurements from the same profile?
What covariance identity should a whitening matrix \(W\) satisfy for predictive covariance \(\Sigma\)?
A transformed profile has unit marginal variances but substantial residual correlation. Is the covariance-adjustment diagnostic complete? Explain.

Solutions. (1) The scores are \(1.5/0.5=3.00\) and \(1.5/3.0=0.500\). The first difference is much more extreme relative to its predictive spread. (2) \(\Phi^{-1}(0.025)\approx-1.96\); the negative sign means the observation is in the lower predictive tail. (3) The marginal CDF transforms describe each observation separately; predictive dependence can still make their scores move together. (4) It should satisfy \(W\Sigma W^\mathsf{T}=I\). (5) No. Unit marginal variances do not show that correlation has been removed; investigate the covariance estimate and transformation, and interpret the profile cautiously.