Troubleshooting Conflicting VPC, pcVPC, and NPDE Results

Why Predictive Checks Can Disagree

A VPC, a prediction-corrected VPC (pcVPC), and NPDE do not ask precisely the same question. A VPC compares observed and simulated outcome distributions, often within time or prediction bins. A pcVPC first rescales observations and simulations to reduce systematic differences in typical prediction across a bin. NPDE assess standardized discrepancies while accounting for the simulated mean and covariance at the observation design. It is therefore possible for one check to look acceptable while another raises concern, even when all three have been calculated correctly.

The practical task is to locate the source of the disagreement before deciding what it implies. A mismatch may reflect conditioning, bin definitions, correction assumptions, finite simulation error, censoring, or dependence among repeated observations. It may also reflect genuine model misfit. The plots and statistics are evidence about model performance only after their simulation and summary procedures have been checked.

A conditional check is not automatically a residual-only plot. If subject-specific predictions differ, conditional simulated outcomes retain that between-subject spread. What can be compared with residual variability is the simulated deviation from each fixed prediction, not the overall spread of all simulated outcomes pooled across subjects.

Start With the Conditioning and the Scale

Before comparing diagnostics, write down what is held fixed and what is simulated: dosing and sampling design, covariates, subject-specific estimates, population random effects, residual error, and any censoring process. Then identify the quantity each display summarizes: pooled outcomes, within-bin quantiles, residuals around predictions, or standardized subject-level discrepancies. Similar labels do not ensure the same conditioning or target quantity.

The variance decomposition makes the conditional distinction explicit. Let \(m_i\) be the fixed subject-specific prediction for one observation, and suppose the simulated outcome is \(Y_i^{sim}=m_i+\epsilon_i\), where the simulated residual \(\epsilon_i\) has mean zero and variance \(\sigma^2\). Across subjects, the variance of the pooled conditional outcomes is

when residuals have mean zero conditional on \(m_i\) and constant variance. This follows by writing \(Y_i^{sim}-E(Y_i^{sim})=(m_i-E(m_i))+\epsilon_i\). Squaring and taking expectations gives the variance of \(m_i\), plus \(\sigma^2\), plus twice the covariance of \(m_i\) and \(\epsilon_i\); that covariance is zero under the stated conditional mean assumption. Thus the conditional residual standard deviation is not the pooled conditional outcome standard deviation.

Binning and Prediction Correction

VPC quantiles depend on bin membership. Time bins that are too wide can combine observations from different parts of a steep concentration profile; bins that are too narrow can contain too few observations for stable quantiles. Data-dependent binning can also make the observed and simulated summaries incomparable if bin boundaries are recalculated differently for each simulated data set. A defensible comparison uses the same design and clearly specified binning rule, and checks whether conclusions persist under reasonable alternative boundaries.

A pcVPC addresses a different issue: observations in one bin may have different typical predictions because of dose, body size, or other covariates. One common proportional correction, for observation \(ij\) in bin \(b\), is

Here \(P_{ij}\) and \(P_{ij}^{sim}\) are the corresponding predictions, and \(P_{ref,b}\) is a stated reference prediction for the bin. Observations and simulations must use a consistent correction rule. This scaling is most interpretable when outcome magnitude is approximately proportional to prediction. It is not a universal remedy for covariate effects, nonlinear scaling, or incorrect structural predictions.

Simulation Precision, Censoring, and Correlation

Predictive-check summaries have sampling variability, and simulated summaries have additional Monte Carlo variability because only a finite number of replicates is generated. For a simple coverage proportion based on \(n\) independent observations, the standard error under coverage probability \(p\) is \(\sqrt{p(1-p)/n}\). Dependence among repeated observations makes this independent-observation formula too optimistic if it is applied to all records as though they were independent. Replicate-level summaries and subject-level structure should be retained when assessing uncertainty.

Censoring needs equivalent care. If concentrations below a limit of quantification are reported as censored rather than as exact values, replacing them with the limit and treating them as ordinary measurements changes the data distribution. Simulations should apply the same censoring and reporting rule as the observations. Checks can then compare quantities such as the fraction censored by time or dose group, while NPDE procedures must use a treatment of censored observations compatible with the model and the available information. Ordinary normal-score discrepancies for invented exact values at the limit do not represent the original censored data.

For repeated measurements, marginal agreement does not guarantee correct within-subject correlation. A VPC can reproduce the distribution at each time while missing the tendency for a subject’s successive residuals to move together. NPDE use the simulated covariance across observations when standardizing the subject’s vector; a misspecified covariance or an unstable covariance estimate can therefore change their results. Conversely, NPDE marginal calibration alone does not establish independence. Examine the sequence and subject-level pattern of discrepancies, and interpret results with the dependence structure in mind.

A Structured Troubleshooting Sequence

When checks disagree, proceed from implementation to interpretation. This avoids treating a difference between summaries as proof that one diagnostic is “right” and another “wrong.”

These checks are complementary, not interchangeable. A raw VPC is direct for the observed outcome scale but can mix prediction levels within bins. A pcVPC can clarify proportional differences across prediction levels but depends on the correction assumption. NPDE provide a numerical, covariance-standardized assessment but can be sensitive to simulation size, covariance estimation, sparse within-subject data, and censoring treatment. No single diagnostic is a universal pass/fail test.

Common Mistakes and Limitations

• Calling residual spread the spread of conditional outcomes. Fixed subject predictions still vary across subjects. Report residual deviations separately from pooled simulated outcome dispersion.
• Assuming a pcVPC is always superior. Correction changes the scale and can be misleading when prediction-proportional scaling is inappropriate. Compare corrected and uncorrected displays.
• Reading one unstable bin as decisive. Sparse bins, extreme quantiles, and too few simulations yield noisy summaries. Inspect bin counts and simulation precision.
• Treating censored values as exact. Artificially setting every censored value equal to the limit changes the analysis target. Replicate the observation and censoring process.
• Equating marginal fit with joint fit. Matching time-specific distributions does not establish that repeated observations have the correct dependence.
• Interpreting all disagreement as model failure. First rule out differences in conditioning, scales, binning, and simulation precision; then assess persistent structured patterns.

A useful report states the conditioning rule, simulation count, binning method, prediction-correction formula, censoring treatment, and how repeated-measure dependence enters the numerical check. These details make it possible to distinguish a robust diagnostic pattern from an artifact of implementation. Even after they are checked, predictive checks have finite data and simulation resolution: they can reveal discrepancies but do not identify a unique model repair.