Normalized Prediction Distribution Errors for PD

Why Normalize Prediction Errors?

A residual is an observed response minus a model prediction, but its size is meaningful only relative to the variability the model expects. In PD data, that variability may differ across observations, and repeated measurements from one individual may be correlated. A residual of 2 response units could be unusual at one time and entirely plausible at another. Normalized prediction distribution errors (NPDEs) address this by using simulated responses from the fitted model to put prediction errors on a common scale.

The target is not merely a residual with a divided-by-standard-deviation adjustment. NPDEs use the simulated predictive distribution, including its estimated variability and, for repeated observations, its dependence structure. After decorrelation, the errors are mapped through their simulated distribution to normal scores. If the predictive model is appropriate and the calculation is well behaved, the resulting NPDEs should be approximately centered at zero, have variance one, and follow a standard normal distribution.

The prediction level must be chosen deliberately. A population-level predictive distribution includes the between-subject variability specified by the model; a conditional predictive distribution may condition on individual information or estimates. The simulations must match the diagnostic question. For example, population simulations should include the appropriate random effects and residual error, while conditional simulations should use the same conditioning information as the predictions being assessed. Mixing these levels can make a sound model appear too variable or not variable enough.

Calculation: Decorrelate, Then Transform

For each individual, simulate \(M\) complete response vectors under the chosen predictive model. Complete vectors matter: simulating each time point independently would discard the correlation the method is intended to account for. Estimate the predictive mean vector \(\hat{\mu}_i\) and covariance matrix \(\hat{V}_i\) from the replicates. With \(M\) simulated vectors, the usual sample covariance uses divisor \(M-1\).

Choose a whitening matrix \(A_i\) such that \(A_i\hat{V}_iA_i^T=I\), where \(I\) is the identity matrix. Apply the same matrix and centering to the observed vector and every simulated vector. This produces decorrelated coordinates with unit variance under the estimated covariance. A Cholesky-based whitening is one possible choice; the observation order and whitening convention must be kept fixed when comparing results.

For each coordinate \(j\), estimate the marginal cumulative distribution of the transformed simulations, denoted \(F_{ij}^{*}\). Map the observed transformed value to its percentile \(u_{ij}=F_{ij}^{*}(q_{ij})\), then apply the inverse standard-normal cumulative distribution function \(\Phi^{-1}\). In practice, a finite-simulation plotting-position or rank convention is used so that the percentile is strictly between zero and one; ties require a stated convention. The same procedure is applied consistently across observations.

The logic is that, under a correct predictive distribution, a value’s percentile is uniformly distributed between zero and one. Applying \(\Phi^{-1}\) to a uniform percentile gives a standard normal variable. The simulated covariance adjustment first addresses dependence among coordinates; the empirical-distribution transformation then calibrates each coordinate’s marginal scale and shape. Thus NPDEs do not require the original prediction errors themselves to be normally distributed.

Worked Examples

Assumptions, Diagnostics, and Failure Modes

NPDEs are diagnostics of a predictive model, not a device that can rescue an inappropriate simulation design. Their interpretation depends on several conditions:

• Matched simulations: Replicates use the observed dose history, sampling times, covariates, baseline information, and other design features relevant to the prediction target.
• Correct variability level: The simulations include the random effects and residual variation appropriate to population- or conditional-level predictions.
• Adequate simulation and covariance estimation: There are enough replicate vectors to estimate the predictive distribution and a nonsingular covariance matrix. A nearly singular covariance can make whitening unstable.
• Dependence is represented: Each replicate preserves the modeled within-subject dependence. When summaries across subjects are interpreted as a distribution, subject-level contributions should also be considered for dependence or clustering.
• Model fitting is acknowledged: Common implementations simulate from fitted parameter estimates. This plug-in approach does not automatically include uncertainty in those estimates, and fitting the model to the same data can affect diagnostic calibration.

Inspect more than the mean. A mean away from zero can indicate systematic overprediction or underprediction, although the sign depends on the residual convention used in constructing the error. A standard deviation substantially above one suggests errors are more dispersed than the predictive model allows; a value substantially below one suggests predictions may be too variable. Skewness, heavy tails, or multiple modes can remain even when the mean and standard deviation look acceptable. Normal quantile plots and histograms help reveal such departures.

Also inspect NPDEs against time, predicted effect, concentration, dose, baseline, and relevant covariates. A pooled distribution can appear well centered while errors drift over time or have opposite biases in subgroups. Such a pattern can point to structural or covariate misspecification, an incorrect variability model, or a mismatched conditioning level. These plots are diagnostic signals, not proof of a particular cause; interpret them alongside the model and other predictive checks.

How NPDEs Relate to Other Diagnostics

Raw residuals retain response units and are useful for seeing the direction and size of discrepancies, but their scale is not directly comparable across observations with different predictive variability. Variance-standardized residuals divide by an estimated standard deviation and can be easier to compare, but depend on the adequacy of that variance estimate and do not generally correct the full predictive distribution or repeated-measure correlation. NPDEs use simulation to address those features, at the cost of additional computation and dependence on the simulation algorithm.

Predictive-check summaries, such as comparing simulated and observed response distributions, are complementary: they can show whether important features of the data are reproduced, but do not assign each observation a normal-score residual. Conversely, NPDEs provide observation-level scores but can hide localized problems if only their pooled distribution is examined. Use the methods together rather than treating one as a substitute for all others.

NPDEs also have limits. Finite simulation introduces Monte Carlo noise, especially in the tails, where a small change in estimated percentile can cause a noticeable change in \(z\). A large number of simulations, stable covariance estimates, and consistent rank handling improve reliability. Whitening is a linear decorrelation step; in a strongly non-normal joint distribution, zero correlation after whitening does not guarantee statistical independence. Therefore, approximate standard normality is a diagnostic expectation under the specified predictive construction, not a universal guarantee for every model and sample size.