Predictive Checks Across Multiple Endpoints

Why Check Endpoints Separately?

A pharmacometric model may describe several outcomes measured on the same subjects, such as drug concentration and a continuous biomarker. A model can reproduce one outcome well and miss another; even when both are adequately reproduced, their numerical scales may be very different. A predictive check across endpoints must therefore do two things at once: assess each endpoint in a meaningful way and preserve the fact that outcomes from the same subject may be related.

The basic strategy is to make endpoint-specific predictive summaries. For each outcome, compare the observed data with replicate data generated under the fitted model, using summaries appropriate to that outcome and its sampling schedule. Interpret the resulting checks together, but do not combine raw values from different endpoints as if they had a common scale.

Constructing Endpoint-Specific Checks

For each endpoint, first decide what feature of the data the check is meant to assess. A continuous concentration endpoint might be summarized by medians and percentiles within time bins; a continuous biomarker might be summarized by its mean or by percentiles within clinically meaningful time windows. The statistic should reflect the question and the outcome’s distribution. The observed and simulated data must then be summarized in exactly the same way.

Suppose there are \(R\) simulated replicate data sets. For endpoint \(k\), let \(T_k^{obs}\) be the observed summary and \(T_{kr}\) the summary from replicate \(r\). The empirical predictive distribution of \(T_{kr}\) describes the range of summaries produced by the fitted model under the chosen simulation scheme. Comparing \(T_k^{obs}\) with this distribution makes the check interpretable in that endpoint’s own units.

The simulation scheme matters. Simulations should reproduce the study design relevant to the check, including doses, observation times, and the records included in each summary. If an endpoint has missing observations, the simulated data should be handled consistently with the observed data and the intended estimand. A comparison can be misleading if, for example, observed summaries use only available records while simulated summaries use a different pattern of records.

When several endpoints are measured on the same subjects, joint simulation is preferable: each replicate should generate all endpoints for the same simulated subjects, using the model’s shared subject effects and any modeled dependence between outcomes. Simulating each endpoint independently can preserve its marginal distribution while failing to reproduce cross-endpoint association. That failure matters when the question concerns the pattern of outcomes within subjects or a joint summary.

Shared Subjects and Cross-Endpoint Dependence

If two endpoints are measured on the same subject, their observations are generally not independent. They may share subject-specific effects, such as an individual’s underlying disease severity, or have correlated residual variation. A model may represent these links explicitly, or it may leave some dependence unmodeled. In either case, interpretation must distinguish a marginal check of each endpoint from a joint check of their relationship.

A separate endpoint-specific VPC or predictive summary can assess whether each endpoint’s marginal behavior is plausible. It does not, by itself, show that the model reproduces the association between endpoints. To examine that association, define a joint statistic—such as the within-subject covariance of appropriately scaled residuals—and calculate it for both the observed and jointly simulated data. The statistic and its interpretation must account for the pairing of outcomes by subject.

Different Scales and Meaningful Comparisons

Endpoints may have different units, ranges, and variability. A concentration in mg/L and a biomarker in units cannot sensibly be compared by subtracting their raw values or pooling them into one mean error. First compare each observed summary with its own endpoint’s predictive distribution. If a common display is useful, scale deviations relative to endpoint-specific predictive variation or show endpoint-specific predictive quantiles; explain clearly what the scale represents.

Standardization can help compare the relative extremeness of summaries, but it does not make unlike scientific outcomes interchangeable. A deviation of two predictive standard deviations for one endpoint does not necessarily have the same clinical importance as the same standardized deviation for another. Clinical relevance depends on the endpoint, its units, and the consequences of the discrepancy, not only on its position in a predictive distribution.

The reason standardization removes units is direct. If a summary \(T\) is transformed to \(T^*=a+bT\), with \(b>0\), then its predictive mean and standard deviation become \(a+b\bar T\) and \(bs_T\). Consequently,

The standardized deviation is unchanged by a positive change of units, such as converting a concentration from one unit to another. It remains a summary of relative predictive discrepancy, not a universal measure of clinical importance.

Multiplicity: Many Checks, More Opportunities for an Anomaly

A multi-endpoint assessment often includes several endpoints, time windows, bins, and summaries. Even when a fitted model generates data consistent with the simulation assumptions, one of many inspected summaries may look unusually extreme by chance. This is a multiplicity issue: the more checks a reviewer scans, the greater the opportunity to notice at least one apparent anomaly.

For illustration, if four independent checks each had a 5% chance of crossing a prespecified warning threshold under the model, the probability that at least one crosses it would be

This calculation assumes independent checks and a threshold with a genuine 5% per-check false-alarm probability. Predictive plots are often dependent, and a visual flag is not automatically a formal test, so 18.55% is an illustration of the multiplicity principle—not a calibrated error rate for a particular set of plots.

A practical response is to specify key endpoints and summaries in advance, distinguish planned checks from exploratory ones, and look for coherent patterns rather than treating every isolated point as decisive. If a single overall assessment is needed, define a joint discrepancy statistic and evaluate that statistic using joint simulations. Do not simply pool raw endpoint values or apply a correction without considering the dependence structure and the purpose of the check.

Diagnostics, Alternatives, and Limitations

A multi-endpoint check is difficult to interpret when the simulation design or summary definitions differ across endpoints in unacknowledged ways. Check that observed and simulated records are matched by endpoint and relevant design features, that units and transformations are correct, and that all simulated summaries use the same rules as the observed summaries. Sparse data, very few simulations, or highly skewed predictive summaries can also make apparent bounds or standardized deviations unstable.

The main alternative to a multi-endpoint display is a set of endpoint-specific displays. These are often clearer when endpoints have very different scales or sampling schedules, and they make each endpoint’s predictive distribution easier to inspect. A combined panel can make patterns across endpoints easier to see, but should retain separate axes or other explicit scale labels. A joint statistic can summarize a prespecified cross-endpoint feature, but may hide which endpoint drives the result. No single format replaces careful endpoint-specific interpretation.

Predictive checks depend on the model, the simulation conditions, and the summaries selected. Agreement in a chosen set of summaries does not establish that every feature of the joint data distribution is adequate. Conversely, a visually unusual point does not on its own identify the cause of a discrepancy. These checks are diagnostic evidence, to be interpreted alongside the study design and other model-evaluation information.