Why Sampling Design Matters for NPDEs
Normalized prediction distribution errors (NPDEs) compare observed concentrations with the model’s predictive distribution while accounting for the dependence among repeated observations. In a sparse design, an individual may contribute only one or two concentrations. In an unbalanced design, individuals may be sampled at different times, or contribute different numbers of observations. These features change what an NPDE can diagnose and how precisely it can be estimated; they do not automatically make the diagnostic invalid.
The key requirement is that the predictive simulations represent the design being evaluated. For each individual, simulations should use that individual’s relevant dose history, covariates, and observation times. The comparison is then between the observed data and predictions for the same conditions, rather than predictions for an imagined common schedule. The earlier material on deriving NPDEs and decorrelation establishes the general calculation; here the emphasis is on what changes when the design is sparse or uneven.
The number of records per individual and the number of predictive simulations are different quantities. A subject with one observation has a one-dimensional predictive distribution; the model can still provide many simulated values from it. By contrast, a subject with several observations requires a joint predictive distribution, including the covariance among those observations. A large number of simulations can estimate that distribution more precisely, but cannot create information about model behavior in parts of the design that were never observed.
Calculation Under Sparse Designs
For one observation, the NPDE reduces to a normal-score transformation of the predictive cumulative distribution. If the model predicts a continuous random variable \(Y\) with cumulative distribution function \(F\), the prediction error for observed value \(y\) is \(z=\Phi^{-1}\{F(y)\}\), where \(\Phi\) is the standard normal cumulative distribution function. If \(F\) is correctly specified and continuous, the probability integral transform \(F(Y)\) is uniform on \((0,1)\); consequently, \(\Phi^{-1}\{F(Y)\}\) has a standard normal distribution.
With multiple observations, a vector of raw deviations cannot generally be treated as independent components: concentrations from the same individual may be predictively correlated. For a multivariate normal predictive distribution with mean vector \(\boldsymbol{\mu}_i\) and covariance matrix \(\boldsymbol{\Sigma}_i\), let \(\mathbf{L}_i\) be a Cholesky factor satisfying \(\boldsymbol{\Sigma}_i=\mathbf{L}_i\mathbf{L}_i^\mathsf{T}\). The decorrelated prediction error is \(\mathbf{d}_i=\mathbf{L}_i^{-1}(\mathbf{y}_i-\boldsymbol{\mu}_i)\). Under that multivariate normal model, \(\mathbf{d}_i\) has independent standard normal components. For non-normal predictive distributions, the simulated data are used to estimate the corresponding marginal normal-score transformations after decorrelation; a covariance transformation alone does not guarantee independence for an arbitrary distribution.
In practice, the predictive mean, covariance, and (where needed) marginal transformations are estimated from the simulated vectors. The dimension of the covariance matrix is the number of observations for that individual, not the number of individuals. Sparse individual records therefore often mean a low-dimensional calculation, while unbalanced records mean that different individuals have different-dimensional calculations. A shared visit index, such as “the third sample,” does not necessarily represent a shared time: the actual times and design must inform the prediction.
Worked Example: One Observation Per Individual
Suppose a hypothetical individual has one concentration measured at a specified time. Conditional on the individual’s design, the model predicts \(Y\sim N(10,4)\), where concentration is measured in mg/L and 4 is the predictive variance in \((\text{mg/L})^2\). The observed concentration is 12 mg/L. Because the predictive standard deviation is \(\sqrt{4}=2\) mg/L, the standardized prediction error is
For this normal predictive distribution, \(F(12)=\Phi(1)\approx0.8413\), so \(\Phi^{-1}\{F(12)\}=1.00\). The observation is one predictive standard deviation above the conditional mean. It is one NPDE contribution, not evidence by itself that the model systematically underpredicts concentrations. With only one record, this individual provides no within-individual information about the time course or covariance between visits.
Worked Example: Two Unevenly Timed Observations
Consider a hypothetical individual sampled at 1 hour and 12 hours after a dose. Suppose the predictive mean vector is \((8,4)^\mathsf{T}\) mg/L and the covariance matrix, in \((\text{mg/L})^2\), is
The observed concentrations are \((10,4.8)^\mathsf{T}\) mg/L, so the residual vector is \((2,0.8)^\mathsf{T}\) mg/L. A Cholesky factor is
Solving \(\mathbf{L}\mathbf{d}=(2,0.8)^\mathsf{T}\) gives \(2d_1=2\), hence \(d_1=1\), and \(0.6d_1+0.8d_2=0.8\), hence \(d_2=(0.8-0.6)/0.8=0.25\). Under the stated multivariate normal predictive model, the decorrelated errors are therefore 1.00 and 0.250. The second value is not obtained by simply dividing the second residual by its marginal standard deviation: that would give 0.8. The decorrelation step accounts for predictive association with the first observation. Since the observations are at different times, the result applies to this pair of times and this individual’s predictive distribution; the two component positions should not be assumed to mean the same times for every other individual.
What Can Be Pooled Across Individuals?
If the model is adequate and the predictive distribution is correctly specified, NPDEs should be approximately standard normal over repeated data sets under that model. This gives familiar checks of location, spread, distributional shape, and association with time or covariates. Sparse sampling mainly reduces the information available for those checks. A subject with one sample contributes to the overall distribution of errors, but cannot reveal whether a within-subject pattern rises or falls across visits.
Unbalanced designs create an additional interpretation issue. If NPDEs are pooled by observation, an individual with many records contributes more values than an individual with one record. The pooled distribution is therefore observation-weighted, not an equal-weight summary of individuals. That can be appropriate when the target is performance across recorded observations, but it should be stated. It is not automatically a summary in which each subject counts equally. When sampling times differ, an overall histogram can also hide a time-localized problem: excess positive errors at one time may be diluted by errors near zero at other times.
Worked Example: Unequal Precision at Different Times
Suppose a hypothetical model predicts a concentration at an early time with mean 8 mg/L and standard deviation 2 mg/L, and at a later time with mean 4 mg/L and standard deviation 1 mg/L. One individual sampled only early has an observed value of 12 mg/L; another sampled only late has an observed value of 6 mg/L. Their raw residuals are 4 mg/L and 2 mg/L, respectively, but the normal-score errors are
Both observations are two conditional predictive standard deviations above their respective means. Comparing raw residuals alone would misleadingly suggest that the early observation is more discrepant simply because its concentration scale is higher. In a larger hypothetical data set, however, one positive NPDE at each time would not establish a time trend. The pattern must be assessed over enough independent individuals and with attention to how many observations were made at each time.
Assumptions, Diagnostics, and Alternatives
NPDE interpretation depends on the predictive simulations being generated under the model and at the observed design. Important conditions include using the correct dose and covariate information, representing the relevant residual and interindividual variability, and handling the observation process consistently. The usual reference to a standard normal distribution is a model-based calibration target, not a guarantee that a small finite sample will look normal. Parameter-estimation uncertainty can also matter; its treatment is a separate modeling choice and should not be silently assumed to be included in simulations.
Useful diagnostics for sparse or unbalanced designs include:
- Check design coverage. Tabulate the number of subjects and observations over time ranges, and identify design regions represented by very few records.
- Check conditioning. Confirm that simulations use the observed individual times and relevant dosing and covariate history. A correct model simulated at the wrong schedule is not a valid comparison.
- Check interpretation by time and design. Plot or summarize NPDEs against actual time and, where useful, by sampling schedule or observation count. Do not treat a visit number as a time when schedules differ.
- Check dependence and influence. Repeated errors from one individual are not independent evidence. A few densely sampled individuals can strongly influence observation-level summaries.
- Check simulation adequacy. Monte Carlo estimates of predictive distributions can be noisy, especially in tails. Increase simulation precision or assess rank stability before interpreting an extreme empirical score.
A visual predictive check (VPC) offers a complementary view: it compares observed and simulated concentration summaries across time bins, while NPDEs express predictive discrepancies on a normalized scale. Under sparse sampling, VPC quantiles can be unstable in bins with few observations, while an overall NPDE distribution can obscure where a discrepancy occurs. Neither summary repairs a lack of design support. Univariate prediction errors can be useful when each individual contributes one observation, but they do not assess the joint predictive behavior of repeated measurements. When multiple records are present, NPDEs aim to account for their predictive dependence; the interpretation still depends on adequate simulations and appropriate stratification.
A common mistake is to argue, “There are too few observations per subject to calculate NPDEs, so I will pool raw residuals across everyone.” That confuses the number of records per subject with whether the predictive distribution can be simulated. One observation can be compared with its one-dimensional predictive distribution, and two or more can be compared with the corresponding joint distribution. The correction is to simulate at each subject’s actual design, calculate the appropriate predictive error, and then interpret the resulting collection with its unequal weighting and limited diagnostic power in mind.
The main limitation is not that sparse-data NPDEs are inherently invalid; it is that they cannot answer questions unsupported by the design. If almost no individuals are observed late after dosing, a reassuring pooled diagnostic does not establish late-time performance. If each person contributes a single observation, the data cannot show within-person correlation patterns. Moreover, approximate normality of a pooled set is only one aspect of predictive adequacy: it can coexist with structured errors in a sparsely represented subgroup or time region.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A subject has one observation, and its continuous predictive CDF at the observed concentration is 0.9772. What is the normal-score prediction error, approximately?
- Why should a subject’s predictive simulations use that subject’s actual observation times?
- Two subjects have one observation each. Their predictive means and standard deviations are 5 mg/L and 1 mg/L for the first, and 10 mg/L and 2 mg/L for the second. Their observations are 7 mg/L and 14 mg/L. Compare their normal-score errors.
- In an unbalanced data set, why might a pooled observation-level NPDE histogram not give each individual equal influence?
- What can and cannot be learned about within-subject behavior when nearly every individual contributes only one concentration?
Solutions. (1) \(\Phi^{-1}(0.9772)\approx2.00\), so the observation is about two predictive standard deviations above its conditional distribution. (2) The predictive distribution depends on time and may depend on the dosing and covariate history; simulating another schedule would compare the observation with the wrong conditions. (3) The first error is \((7-5)/1=2\); the second is \((14-10)/2=2\). Both are two standard deviations above their respective means. (4) Each observation contributes a value, so subjects with more observations contribute more values and can dominate the pooled distribution. (5) Each observation can be assessed against its own predictive distribution, but one record per person cannot reveal within-person changes across time or estimate that person’s observed correlation pattern.