Deriving Normalized Prediction Distribution Errors

From Predictive Simulations to NPDEs

A visual predictive check compares observed and simulated summaries, such as concentration quantiles within time bins. A normalized prediction distribution error (NPDE) instead assigns a standardized discrepancy to an observed concentration profile using its simulated predictive distribution. The transformation has two parts: first, determine where the observation falls in the simulated distribution; second, for a repeated-measures profile, remove the correlation among the resulting normal scores.

The second part matters because concentrations measured repeatedly in one individual are not independent. A profile that is generally high relative to simulations may produce several high marginal scores together. Treating those scores as independent would misrepresent the joint predictive behavior. NPDEs are intended to put profile-level predictive discrepancies on a comparable, decorrelated scale; they do not replace examination of the observed data, model assumptions, or predictive simulations.

The intended predictive distribution is the model distribution for a new profile under the relevant design: the individual’s recorded dose history, observation times, and covariates are held to their observed values, while the model simulates the sources of variation included in the prediction. For a population prediction, these typically include between-individual variability and residual error. Simulated profiles and the observed profile must be compared on the same scale and under the same prediction target.

For a scalar concentration \(y\), let \(F(y)\) denote its predictive cumulative distribution function. If the predictive distribution is continuous and correctly specifies the data-generating distribution, then \(F(Y)\) is uniformly distributed on \((0,1)\). Applying the standard-normal quantile function \(\Phi^{-1}\) therefore gives a standard-normal score. In practice, \(F\) is approximated from a finite number of simulated concentrations.

Empirical Ranks and Normal Scores

Suppose \(N\) simulated concentrations are available at one observation time, and \(b\) of them are strictly less than the observed concentration \(y\). One useful finite-simulation plotting-position convention is

This convention keeps the estimated percentile strictly between zero and one, so the inverse normal quantile is finite. It also makes the calculation explicit when the observed value is not tied with a simulated value. Other implementations may use a different rank convention; the convention should be specified and used consistently. With tied or rounded concentrations, a tie rule is needed, such as assigning midranks or using randomized ranks. The continuous-distribution argument does not make tied ranks exactly uniform.

A low percentile produces a negative normal score, while a high percentile produces a positive score. A normal score near zero corresponds to a percentile near 0.5. A positive score does not mean that the observed concentration is absolutely high; it means that the observation is high relative to its model-predicted distribution at that design point.

Why Profiles Need Decorrelation

For an individual with \(m\) observed concentrations, calculate one empirical predictive percentile and one normal score for each observation. Collect the scores in the vector \(\mathbf{z}\). Even if every component has approximately a standard-normal marginal distribution, the components need not be independent. Shared individual effects, residual structure, and the pharmacokinetic trajectory can make scores at different times correlated.

Let \(\mathbf{R}\) be the covariance matrix of the marginal normal-score vector under the predictive model. Ideally, the marginal variances are one, so \(\mathbf{R}\) is also a correlation matrix. Estimate this matrix from simulated profiles processed by the same rank and normal-score procedure. If \(\mathbf{R}\) is positive definite, take a Cholesky factorization \(\mathbf{R}=\mathbf{L}\mathbf{L}^{T}\), where \(\mathbf{L}\) is lower triangular. The decorrelated vector is

The justification is direct. Under the model, if the score vector has mean zero and covariance \(\mathbf{R}\), then

The transformation therefore removes the modeled linear covariance and gives unit variances. Under a multivariate normal predictive distribution, it also produces independent standard-normal components. With empirical ranks, estimated covariance, and finite simulations, the result is approximate rather than exact. NPDEs are consequently model-based predictive residuals, not a guarantee that the observed errors are normal or independent.

Practical Calculation and Checks

A defensible calculation follows a fixed sequence. Simulate complete concentration profiles at the observed design points, calculate the empirical percentile for each observed concentration, transform the percentiles with \(\Phi^{-1}\), estimate the covariance of the resulting simulated score vectors, and apply the corresponding inverse Cholesky factor to each observed score vector. The same ordering of observation times must be used for the observed and simulated vectors and for the covariance matrix.

The simulated profiles used to estimate covariance should reflect the same predictive setup as the observations. A correlation matrix borrowed from a different design, a different set of time points, or a different prediction target can produce inappropriate NPDEs. Check that the matrix is symmetric and positive definite and that its time ordering is correct. An estimated matrix that is singular or nearly singular may indicate too few simulated profiles relative to the number of observations, redundant measurements, or numerical problems; its inverse factor can then amplify small errors.

Very extreme percentiles also warrant attention. A finite simulation cannot resolve tail probabilities more finely than its rank resolution, and a plotting-position adjustment does not create information beyond the simulations. More simulations may improve resolution. This is distinct from a model discrepancy: a concentration near the edge of a finite simulated sample can reflect limited Monte Carlo resolution, model behavior, or both.

Comparison, Limits, and a Common Pitfall

A VPC emphasizes how observed and simulated distributions compare across time or strata, often through quantiles and reference envelopes. NPDEs instead produce pointwise, profile-based standardized discrepancies that can be summarized or plotted against time and covariates. Neither is universally superior: VPC summaries retain concentration-scale interpretation and show distributional features, while NPDEs facilitate a common standardized scale and account for modeled within-profile correlation. The introductory material on predictive checks and VPCs provides the context for interpreting simulated-versus-observed summaries; NPDEs are a complementary representation, not a substitute for those checks.

A common mistake is to transform each observed concentration to a normal score and then call those scores NPDEs, without addressing correlation. For example, two correlated scores of \(1\) and \(1\) do not provide the same joint evidence as two independent scores of \(1\) and \(1\). The correction is to estimate the predictive covariance of score profiles and decorrelate using a specified transformation. Another error is to reverse the percentile direction: counting simulations greater than the observation estimates an upper-tail probability, whereas the CDF percentile counts values at or below the observation. Mixing these conventions reverses score signs.

NPDEs also depend on the predictive model and the simulation design. They do not identify which model component caused a discrepancy, and an apparently well-behaved NPDE distribution does not prove that all aspects of the model are adequate. Rank discreteness, ties, covariance estimation, and limited tail resolution matter, particularly with few simulations or many observations per profile.