Predictive Checks for Irregular Observation Schedules

Why Irregular Observation Times Matter

In a pharmacokinetic study, one subject may have samples at 0.5, 2, and 8 hours after a dose, while another has samples at 1 and 4 hours. A predictive check should evaluate the model against the design that produced the observations. If simulations instead place every subject on a common, convenient time grid, the check no longer represents the observed sampling process: it may create outcomes at unobserved times, omit observed times, or change which subjects contribute to a time-bin summary.

Irregular schedules matter in two related ways. First, predictive outcomes depend on the time of measurement, so simulating at the wrong time changes the comparison. Second, a pooled visual predictive check (VPC) summarizes the observations that fall in each time bin. The number and types of subjects contributing to a bin can vary across time. Consequently, pooled summaries can change with schedule composition even when the model’s predictions for each subject or subgroup have not changed. NPDE calculations have a related requirement: the covariance among a subject’s repeated outcomes must correspond to that subject’s actual observation times.

For a typical NLME model, the outcome at time \(t_{ij}\) can be written as \(Y_{ij}=f(t_{ij},\theta,\eta_i)+\epsilon_{ij}\), where \(\theta\) denotes population parameters, \(\eta_i\) the subject-specific random effects, and \(\epsilon_{ij}\) the residual error. A predictive check may simulate the random effects and residual errors according to its chosen unconditional or conditional target. Either way, if the purpose is to compare simulated and observed outcomes under the study design, the simulated outcome for subject \(i\), observation \(j\), is evaluated at that subject’s \(t_{ij}\). The earlier material on unconditional and conditional predictive checks distinguishes the targets; the schedule requirement applies to both.

Simulate at the Observed Times

In replicate \(r\), let \(Y_{ij}^{(r)}\) denote the simulated outcome corresponding to observed record \((i,j)\). A basic design-preserving procedure uses the same \(t_{ij}\) for every replicate, while regenerating the stochastic quantities required by the predictive target. When dose histories or covariates affect the prediction, those inputs should also match the record being simulated. The schedule is retained; the outcome is not copied from the observed data.

A common-grid plot can still be useful for displaying a fitted typical curve, but it is not a substitute for design-preserving simulation in a predictive check. Interpolating observed outcomes onto a common grid is especially risky: interpolation creates values that were not measured and imposes assumptions about the trajectory between measurements. A grid may be used for display after the check, provided it is clear that it does not replace the record-level comparison at actual observation times.

How Schedule Composition Affects a Pooled VPC

A VPC compares observed outcome summaries with the corresponding summaries from simulated data, often within time bins. At a given time bin, the pooled observed quantile is calculated from the observations that actually fall in that bin. Simulated quantiles should be calculated from simulated records at those same subject-specific times and with a comparable binning rule. A pooled quantile is therefore a summary of both the predictive distributions and the records represented in the bin.

If subjects with different covariates, doses, or model-predicted concentrations contribute in different proportions across time bins, the pooled distribution can shift as that composition changes. This need not indicate that any subject’s prediction changed. It can instead reflect which subjects were sampled at that time. The following simplified calculation uses predicted values without residual error to isolate the composition effect.

Prediction correction can help address unequal typical predictions across observations, but it does not automatically make the sampling design irrelevant. In particular, it does not ensure that the same subject or covariate mix contributes to every time bin. Depending on the purpose of the check, useful complements include stratifying by an influential covariate or dose group, displaying the number of observations per bin, and showing the distribution of schedules or sampling times. Any weighting scheme changes the estimand: an observation-weighted summary describes the pooled records, whereas a subject-weighted summary gives subjects more equal influence. State which summary is being shown rather than treating these as interchangeable.

NPDE and the Subject-Specific Time Vector

For NPDE, the simulated outcomes for a subject form a vector ordered by that subject’s observed times. Their covariance depends on the observation times as well as on the model. For instance, two observations close together may have a different modeled residual or predictive correlation from two observations far apart. Thus the covariance matrix used for a subject must correspond to the same times and ordering as the observed outcome vector. Using a common-time covariance for subjects with different schedules can yield incorrectly standardized deviations.

The basic covariance transformation makes this requirement explicit. Suppose a subject’s predictive vector has mean \(\mu_i\) and positive-definite covariance matrix \(\Sigma_i\), both evaluated at that subject’s actual schedule. Let \(L_i\) be a Cholesky factor such that \(\Sigma_i=L_iL_i^\mathsf{T}\). The whitened deviation is \(Z_i=L_i^{-1}(Y_i-\mu_i)\). If \(Y_i\) has the specified multivariate normal predictive distribution, then the transformed vector has mean zero and identity covariance:

This is the idealized covariance step behind NPDE. In practice, NPDE are estimated from simulated replicates, and the transformation includes the appropriate marginal normal-score step; finite simulation size and estimation affect the result. The derivation explains why the covariance must match the subject’s schedule: whitening removes the dependence represented by \(\Sigma_i\), not dependence from some other time vector.

Diagnostics, Alternatives, and Limitations

Several checks help identify schedule-related failures before interpreting a predictive plot or NPDE summary:

• Verify record alignment. For each simulated record, check subject identity, observation index, time, and any design inputs used by the model. A time-ordering or matching error can invalidate both VPC bins and covariance calculations.
• Inspect bin support. Report the number of observed records contributing to each time bin, and examine whether particular covariate, dose, or schedule groups dominate. Sparse bins make quantiles unstable, quite apart from any model problem.
• Compare composition across bins. If the mix of subjects changes over time, assess whether the pooled pattern is consistent with that change. Stratified summaries can help distinguish a within-group discrepancy from a changing pooled mix, though they may be imprecise when strata are small.
• Check the NPDE time vector and covariance. Confirm that the order and dimension of the observed vector, simulated vectors, and covariance matrix agree for every subject. Look for residual trends or remaining dependence in NPDE; a transformation based on mismatched times cannot remove the dependence of the intended predictive distribution.
• Check numerical stability. Very sparse schedules, nearly redundant observations, or too few simulated replicates can make covariance estimates unstable or singular. Investigate the cause rather than silently substituting a covariance from another schedule.

An ordinary pooled VPC is useful for showing how the model predicts the population of observed records, but its summary can be driven by schedule composition. Stratified VPCs make selected design differences more visible at the cost of smaller bin sizes. Prediction-corrected VPCs address variation in typical predictions under their stated construction, but do not by themselves correct a changing mix of subjects. NPDE use within-subject predictive covariance to assess standardized deviations and dependence, but depend on an adequate predictive distribution and reliable simulation-based estimates. None of these displays can compensate for incorrect timing records or an incorrectly specified model.