Why Check Endpoints Jointly?
A model can predict peak concentration and area under the concentration-time curve (AUC) reasonably well when each endpoint is examined alone, yet still give a poor account of how the two occur together. For example, the model may reproduce the proportion of subjects with high peak concentrations and the proportion with high AUC, but predict too few subjects who have both. A joint predictive check evaluates the paired endpoint values, not merely two separate lists of values.
The pairing matters because peak concentration and AUC are calculated for the same subject and may be dependent. A useful check therefore asks two related questions: are the marginal distributions of the endpoints calibrated, and does the model reproduce their dependence and clinically relevant combinations? A joint check complements, rather than replaces, marginal predictive checks.
Set Up the Comparison
Suppose \(n\) subjects each have two endpoints, such as \(C_{\max}\) and AUC. The observed data contain paired values \((C_{\max,i}, \mathrm{AUC}_i)\) for subject \(i\), where \(i=1,\ldots,n\). For replicate \(b\), simulate a dataset of the same size under the model and the relevant design, then calculate both endpoints for every simulated subject. Write the simulated pair for subject \(i\) as \((C_{\max,i}^{(b)}, \mathrm{AUC}_i^{(b)})\).
Before comparing summaries, confirm that the endpoints have consistent definitions in the observed and simulated data. For example, a peak defined as the maximum of sampled concentrations is not necessarily comparable to a model-derived continuous-time maximum. AUC windows, units, dose conditions, sampling designs, and handling of below-quantification or missing values should also match the intended estimand. The compared subjects or population strata should be defined consistently.
A replicate is one simulated dataset, not one simulated endpoint pair considered in isolation. For each replicate, calculate the chosen statistic across its \(n\) subjects. Repeating this process gives a reference distribution for that statistic under the model and simulation setup. This comparison is conditional on the assumptions and components included in the simulation. If, for instance, parameters are held fixed, the reference distribution does not include parameter-estimation uncertainty.
Define both endpoints, their units, the population or subgroup, and the joint feature to check, such as correlation or the proportion above two thresholds.
For every replicate, simulate both endpoints for each subject under the model, retaining subject-level pairing and the relevant design.
Calculate marginal summaries for each endpoint and at least one joint summary. Apply the same definitions to observed and simulated data.
Locate each observed summary among its simulated values, then examine whether marginal and joint results tell a coherent story.
Marginal Calibration and Joint Calibration
Marginal summaries may include means, medians, quantiles, or the proportions within specified predictive ranges. These answer questions about one endpoint at a time. A joint summary describes how endpoints occur together. Common choices include the Pearson correlation of transformed endpoint values, a joint coverage proportion, or a count in a prespecified region such as “high \(C_{\max}\) and high AUC.”
For skewed positive endpoints, a correlation on the log scale may be more informative than a correlation on the original scale. The scale must be stated because a correlation is scale-specific. Correlation summarizes linear association on the chosen scale; it does not describe every feature of the joint distribution. A scatterplot of paired values, with simulated reference summaries alongside it, can help reveal patterns that a single correlation conceals.
Joint coverage can be defined using endpoint-specific predictive ranges. For each subject, label whether each observed endpoint lies within its corresponding range, then count subjects for whom both endpoints are covered. The range construction must be clearly specified: for example, it may be a population-level interval or a subject-conditional predictive interval. The joint proportion should be compared with the same calculation across simulated datasets, not interpreted using marginal coverage alone.
Worked Example: Joint Coverage of Peak and AUC
Consider a hypothetical dataset of 20 subjects. For each subject, an endpoint-specific predictive range has been defined for \(C_{\max}\) and AUC. The observed data have 18 of 20 \(C_{\max}\) values and 16 of 20 AUC values inside their respective ranges. Twelve subjects have both endpoints inside their ranges.
The marginal coverage proportions are \(18/20=0.90\) for \(C_{\max}\) and \(16/20=0.80\) for AUC. Joint coverage is \(12/20=0.60\). For comparison, simulate 1,000 datasets of 20 paired subjects each and calculate joint coverage in every replicate. Suppose the simulated median is 15 of 20, with a central 95% simulation interval from 11 to 19 subjects.
The observed count, 12, lies inside that reference interval. In this hypothetical example, the joint coverage is lower than the simulated median but not unusually low relative to the simulated variation. The marginal results and joint result should all be reported: the fact that the joint count lies within the reference distribution does not erase the lower AUC marginal coverage. Nor would the two marginal coverages alone establish that joint coverage is adequate.
Checking Dependence and Joint Tails
A dependence check compares an observed dependence summary with its replicate distribution. For example, on the log scale the observed Pearson correlation is $$ r_{\mathrm{obs}} = \frac{\sum_{i=1}^{n}(\log C_{\max,i}-\overline{\log C_{\max}})(\log \mathrm{AUC}_i-\overline{\log \mathrm{AUC}})} {\sqrt{\sum_{i=1}^{n}(\log C_{\max,i}-\overline{\log C_{\max}})^2 \sum_{i=1}^{n}(\log \mathrm{AUC}_i-\overline{\log \mathrm{AUC}})^2}}. $$ Calculate the same correlation in each simulated dataset. Compare the observed value with the simulated distribution, while remembering that correlation alone can miss nonlinear association, outliers, or differences in the tails.
Another useful summary is the fraction above two thresholds. Thresholds can be clinically specified, or chosen to address a particular predictive question. If they are intended to probe dependence after accounting for marginal calibration, thresholds based on fixed marginal reference quantiles can be useful. Whatever the choice, state it and apply it consistently. A high-high count based on clinical thresholds and a high-high count based on marginal quantiles answer different questions.
Worked Example: Simulated Dependence Reference
In a hypothetical study of 30 subjects, the observed correlation between log \(C_{\max}\) and log AUC is 0.82. Across 1,000 paired simulated datasets, suppose the median correlation is 0.55 and 24 simulated correlations are at least 0.82.
The observed correlation is substantially above the simulated median. The upper-tail simulation proportion, using the usual plus-one adjustment, is $$ \frac{1+24}{1+1000}=\frac{25}{1001}=0.02498. $$ This indicates that correlations this large or larger were uncommon in this particular simulated reference distribution. It is evidence that the model may understate positive dependence on the log scale, not a probability that the model is true or false. The result should be interpreted alongside the scatterplot, marginal checks, and simulation assumptions.
This is a directional check because the concern is unusually high dependence. If either unusually high or unusually low dependence would be important, define a two-sided discrepancy in advance—for example, distance from the simulated median—and compare the observed distance with replicate distances. Do not select the tail direction only after seeing which one makes the result look most striking.
Worked Example: Joint High-High Tail
Suppose a hypothetical analysis defines “high” as exceeding 5 mg/L for \(C_{\max}\) and 40 mg·h/L for AUC. Among 40 observed subjects, 6 exceed both thresholds. The observed joint-tail proportion is $$ \frac{6}{40}=0.15. $$
In 1,000 simulated datasets of 40 paired subjects, suppose 7 replicates have a joint-tail proportion at least 0.15. The plus-one upper-tail proportion is $$ \frac{1+7}{1+1000}=\frac{8}{1001}=0.007992. $$ The observed high-high fraction is unusual relative to these simulations. Possible explanations include underprediction of joint high exposure, a mismatch in one or both marginal tails, or a problem in the simulation or endpoint definitions. This summary alone cannot distinguish among them; examine marginal exceedance proportions and the paired scatterplot before attributing the discrepancy specifically to dependence.
Reading the Results and Avoiding Failure Modes
A useful interpretation separates marginal and joint evidence. If both marginal distributions are poorly calibrated, a joint discrepancy may simply reflect those marginal errors. If the marginals appear reasonable but the dependence summary or joint-tail frequency differs from simulations, the model may have a problem in the way the endpoints co-vary. These are diagnostic interpretations, not proof of a unique cause.
The replicate distribution also reflects Monte Carlo variability. With \(B\) simulations, an estimated tail proportion is based on only \(B\) simulated values; very small proportions require many replicates to estimate stably. Report the number of replicates and, where the tail result matters, check whether it changes materially with additional simulations. A simulation interval describes variation across replicated datasets under the specified setup; it is not automatically a confidence interval for a model parameter.
Common failure modes include simulating the endpoints separately and pairing them afterward, comparing endpoints measured on different subjects, and changing endpoint definitions between observed and simulated data. Separately simulating each marginal can preserve both marginal distributions while destroying the dependence that the joint check is meant to assess. Another pitfall is relying on one correlation value: the same correlation can accompany different tail behavior, and a few extreme subjects can have substantial influence.
| Approach | Main question | What it may miss |
|---|---|---|
| Separate marginal checks | Does each endpoint have a plausible distribution? | Whether endpoint values occur together as the model predicts |
| Joint predictive check | Are both marginal behavior and selected features of endpoint pairing reproduced? | Features not captured by the chosen summaries |
| One joint statistic, such as correlation | Does a particular dependence summary match? | Other aspects of joint shape, including nonlinear or tail behavior |
The most defensible assessment uses a small set of complementary summaries chosen for the predictive question: marginal calibration for each endpoint, a dependence summary, and a clinically meaningful joint-region summary when appropriate. No finite set of summaries guarantees that the entire joint distribution is correct. A further limitation is that a predictive check can only diagnose components represented in its simulation setup; a discrepancy may arise from the model, design representation, endpoint calculation, or data processing.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What must be preserved when simulating \(C_{\max}\) and AUC for a joint predictive check, and why?
- If the observed marginal coverage proportions are 0.90 and 0.80, what are the general lower and upper bounds for joint coverage?
- An observed dependence statistic is unusual relative to simulations, but both marginal checks also show discrepancies. Why should the dependence conclusion be cautious?
- In 499 simulated datasets, 9 statistics are at least as extreme as the observed statistic in a prespecified upper-tail check. Calculate the plus-one simulation tail proportion and interpret it.
Solutions. (1) Preserve the pairing of endpoints within each subject in each replicate; otherwise the simulated dependence, including joint-tail behavior, can be changed or erased. (2) The lower bound is \(\max(0,0.90+0.80-1)=0.70\), and the upper bound is \(\min(0.90,0.80)=0.80\). (3) A joint discrepancy may partly follow from miscalibrated marginals, so it cannot be attributed to dependence alone without examining how marginal errors contribute. (4) The plus-one proportion is \((1+9)/(1+499)=10/500=0.0200\). Under this simulation setup, an upper-tail value at least this extreme was uncommon; this is not the probability that the model is true or false.