Why Replicate Data Under Competing Mechanisms?
A mechanistic model can fit observed measurements reasonably well and still fail to reproduce a feature that matters biologically. For example, it may predict biomarker peaks too early, omit a rebound after an initial response, or underestimate differences between subjects. Posterior predictive checks make such questions explicit: if a candidate mechanism and its fitted parameter uncertainty generated new data under the study’s design, would those data show features like the ones observed?
A posterior predictive check is a model-based diagnostic, not a direct test that establishes which biological explanation is true. Its value depends on choosing a discrepancy that represents the feature of interest, simulating the full observation process, and interpreting the result in light of the check’s limitations.
In a QSP application, a replicated data set should generally use the observed subjects, dose regimens, sampling times, and measurement definitions. It should also include the assumed residual error, subject-level variation, censoring, and other relevant observation features. For a hierarchical model, replicated subject effects must be generated according to the model’s stated population and conditioning assumptions; they should not be inadvertently omitted or replaced by fixed fitted values.
A discrepancy measure \(T(y)\) summarizes a feature to be checked. Examples include the time of the observed maximum, a defined rebound magnitude, or the sample standard deviation of subject-level exposure. The observed value \(T(y)\) is compared with values \(T(y^{\mathrm{rep}})\) from replicated data. A useful check asks a focused question, such as whether a mechanism can reproduce the observed rebound, rather than treating every graphical difference as a separate formal test.
Simulation and Interpretation
For each replication, draw \(\theta^{(b)}\) from the posterior for mechanism \(M\), then draw \(y^{\mathrm{rep},(b)}\) from the model’s predictive distribution at that parameter value. Calculate the same discrepancy from observed and replicated data. A one-sided posterior predictive tail proportion for a large-value discrepancy is
where \(B\) is the number of replications and \(I\{\cdot\}\) equals 1 when its condition holds and 0 otherwise. A small value indicates that replicated data rarely reach or exceed the observed discrepancy under this particular check. For a discrepancy where either extreme is concerning, compare the observed value with the simulated distribution or use a clearly defined two-sided discrepancy. There is no universal tail direction: it depends on the scientific question and on how \(T\) is defined.
The basic justification is the predictive integral: uncertainty about \(\theta\) is averaged over the posterior, and the observation model then generates replicated outcomes. Monte Carlo replication approximates this integral by averaging across posterior draws and simulated data. For a tail proportion based on \(B\) independent replications, its approximate Monte Carlo standard error is
This expression follows from the standard error of the mean of \(B\) independent indicator variables, each with estimated success probability \(\hat{p}_{\mathrm{ppc}}\). It describes simulation noise, not uncertainty about the scientific conclusion. If posterior draws are highly correlated, the effective number of independent draws is smaller, so this simple calculation can understate simulation error.
Targeted Checks for Mechanistic Features
Before simulating, specify the feature and how it will be measured. A peak-time discrepancy must use a prespecified time window and the actual observation schedule. If samples are collected every two hours at 0, 2, 4, 6, 8, 10, and 12 hours, a peak defined as the time of the largest measured value must lie on that grid. A model’s continuous-time latent peak may be examined separately, but it is not the same statistic as the peak in the measured data.
For rebound, define the time windows and calculation in advance. For example, the rebound magnitude could be the maximum measured biomarker value from 8 through 16 hours minus the minimum value from 2 through 6 hours. Applying the same definition to observed and replicated data ensures that noise, timing, and sampling affect the comparison in comparable ways. A different biological definition might use a return toward baseline or a change from nadir; the chosen statistic should match the claim being assessed.
For between-subject variation, calculate the same subject-level outcome in the observed and replicated data, then summarize its spread. If the outcome is area under the curve (AUC), for instance, compare the sample standard deviation of subject AUCs, not a standard deviation computed from all repeated measurements as though they were independent subjects.
Worked Example: Is the Observed Peak Time Reproducible?
Suppose a hypothetical study measures a biomarker at 0, 2, 4, 6, 8, 10, and 12 hours. Its largest measured value occurs at 6 hours. For one candidate mechanism, 400 posterior predictive data sets are generated using the same schedule and measurement model. Their measured peaks occur at 4 hours in 50 data sets, 6 hours in 230, 8 hours in 100, and at one of the other scheduled times in 20.
The categories are consistent with the schedule: every reported time is a scheduled measurement, and the remaining 20 replications have peaks at 0, 2, 10, or 12 hours. The observed peak at 6 hours is within the central range of the replicated peak times. In particular, 50/400 = 0.125 of replicated peaks are at 4 hours, 230/400 = 0.575 at 6 hours, and 100/400 = 0.250 at 8 hours. Thus the model commonly produces a peak at the observed time, although this alone does not establish that the mechanism is biologically correct.
If the question is whether the model predicts peaks at least as late as observed, and \(n\) of the 20 remaining peaks are at 10 or 12 hours, the tail proportion is (230 + 100 + n)/400. That one-sided result is not a general goodness-of-fit score; it answers only the stated late-peak question. The check would be invalid if the replicated peaks included times such as 5 or 7 hours, because those cannot be observed maxima on this schedule.
Worked Example: Checking a Rebound Discrepancy
In a hypothetical study, rebound is defined as the maximum measured biomarker value from 8 through 16 hours minus the minimum measured value from 2 through 6 hours. The observed rebound is 12 units. For a large-rebound check, two mechanisms are simulated with 500 replicated data sets each, using the same subjects, sampling schedule, and measurement assumptions.
Under mechanism A, 420 simulated rebounds are at most 8 units, 65 are 9–11 units, 12 are 12–14 units, and 3 are greater than 14 units. The number at least as large as the observed 12 units is 12 + 3 = 15, so
Mechanism A rarely produces a rebound at least as large as the observed one under this check. The approximate Monte Carlo standard error is \(\sqrt{0.0300(0.9700)/500}=0.00763\), so the estimated tail proportion has non-negligible simulation uncertainty.
Under mechanism B, the corresponding counts are 160, 170, 125, and 45. Its tail proportion is \((125+45)/500=0.340\). Mechanism B produces the observed-sized rebound more often in these replications. This comparison is evidence about reproduction of the rebound feature, not a probability that mechanism B is correct. Other targeted checks may reveal failures for either mechanism.
Worked Example: Checking Between-Subject Variation
A hypothetical study has 10 subjects, and the observed sample standard deviation of their AUC values is 18 mg·h/L. For each mechanism, 400 replicated studies are generated with 10 subjects each, including the model’s subject-level variation and measurement process. The same sample standard deviation is calculated from each replication.
For mechanism A, 230 replicated standard deviations are below 10 mg·h/L, 130 are 10–14, 28 are 15–17, 10 are 18–22, and 2 exceed 22 mg·h/L. There are 12 values at least 18 mg·h/L, giving \(12/400=0.0300\). The observed variability is unusual in the upper direction for this model.
For mechanism B, the corresponding counts are 30, 100, 90, 130, and 50. There are \(130+50=180\) replicated standard deviations at least 18 mg·h/L, giving \(180/400=0.450\). Mechanism B reproduces or exceeds the observed spread much more often in this comparison. The computation is meaningful because each replication contains 10 subjects and the discrepancy is calculated across subjects, matching the observed summary.
Comparing Mechanisms and Diagnosing Problems
Posterior predictive checks are most informative when the candidate mechanisms are compared under the same observed data definitions and design. Use the same sampling times, outcome transformations, and measurement-error assumptions where those are shared scientific assumptions. Each mechanism has its own posterior parameter distribution, but the discrepancy must be calculated identically. This makes differences in predictive behavior interpretable as consequences of the candidate models and their fitted uncertainty rather than changes in the checking procedure.
| Approach | Main question | How it differs from a posterior predictive check |
|---|---|---|
| Residual-pattern diagnostics | Do fitted residuals show systematic structure? | They inspect deviations relative to fitted values; replicated data can assess whether an overall feature is reproduced under parameter and observation uncertainty. |
| Predictive cross-validation | How well does a model predict data withheld from fitting? | Posterior predictive checks reuse the observed data to form the posterior, so they are diagnostics of fit rather than an out-of-sample prediction score. |
| Information criteria | How do models compare using a fit-and-complexity summary? | A targeted check asks whether a particular observed feature is reproduced; it does not reduce all fit and complexity considerations to a single ranking. |
These approaches answer different questions and can be used as complementary evidence. A mechanism that passes one targeted check may fail another. Conversely, an unusual tail proportion can result from a poorly chosen discrepancy, an unrepresentative observation model, or Monte Carlo error, not only from an incorrect biological mechanism.
Useful diagnostics include plotting observed and replicated trajectories on the same sampling grid, checking posterior simulation and numerical solver behavior, and examining the distribution of each targeted discrepancy rather than reporting only a tail proportion. Investigate whether a result changes materially with reasonable alternative definitions of the feature. If many discrepancies are inspected, distinguish prespecified biological targets from patterns noticed only after viewing the results; a striking result among many exploratory checks warrants cautious interpretation.
Specify the time window, subjects, units, and calculation before comparing observed and replicated outcomes.
Draw parameters from the mechanism-specific posterior and simulate the study design, subject variation, and observation process.
Locate the observed discrepancy in the replicated distribution, report simulation precision, and state exactly what the check does and does not imply.
A frequent mistake is to treat a posterior predictive tail proportion of 0.03 as a formal 3% probability that the mechanism is wrong. The correction is that 0.03 estimates the fraction of replicated outcomes at least as extreme as the observed discrepancy under a particular model, posterior, and statistic. It does not assign probability to the mechanism, and it is not generally a frequentist p-value with a uniform null distribution. Another common error is to simulate latent trajectories only while comparing them with noisy measured data. Replications must include the observation process if the observed statistic is based on measurements.
Posterior predictive checks also have limitations. Because the observations help form the posterior and are then compared with data generated from that posterior, these checks can be conservative and may not expose every model defect. A broad discrepancy can conceal a mechanistically important mismatch, while an excessively narrow or data-selected discrepancy can exaggerate one. Checks assess the features actually examined; they cannot validate every aspect of a complex QSP model.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A biomarker is measured only at 0, 3, 6, and 9 hours. Can a replicated measured peak time of 7 hours be included in a check that uses the largest measured value? Explain.
- For 300 replications, 18 rebound discrepancies are at least as large as the observed value. Calculate the estimated upper-tail proportion and its approximate Monte Carlo standard error.
- Why must replicated studies include subject-level variation when the observed discrepancy is the sample standard deviation of subject AUC?
- A posterior predictive tail proportion is 0.02. State a correct interpretation and one interpretation that would be incorrect.
Solutions. (1) No. The largest measured value can occur only at a scheduled time: 0, 3, 6, or 9 hours. A 7-hour peak could be a separately defined latent continuous-time peak, but it is not a measured peak for this schedule. (2) The estimate is \(18/300=0.0600\). Its approximate Monte Carlo standard error is \(\sqrt{0.0600(0.9400)/300}=0.0137\). (3) The observed spread is across subjects; omitting subject-level variation changes the distribution of the statistic being checked and no longer represents the assumed study process. (4) Correct: under this model and posterior, about 2% of the simulated discrepancies were at least as large as the observed one, for the specified check. Incorrect: there is a 2% probability that the mechanism is true.