Patient-Level Posterior Predictive Checks

Why Check the Fit for One Patient?

A patient-specific forecast can be useful even when the model does not reproduce the patient’s observed concentration–time pattern particularly well. That distinction matters: a forecast is a distribution of possible concentrations, whereas a fit check asks whether the model, after learning from this patient’s measurements, can generate data resembling those measurements. Patient-level posterior predictive checks make this comparison explicit. They can reveal an observation or a pattern that deserves clinical or modeling attention, but they do not by themselves establish why the mismatch occurred.

The check uses the patient’s recorded concentrations, their sampling times, and the dosing history used in the individual fit. It also uses the fitted model’s uncertainty about individual parameters and, where included, its residual or assay variability. The central comparison is between the observed concentration–time course and replicated courses generated under those same conditions.

Definition: Let $y_i$ denote the vector of observed concentrations for patient $i$, and let $t_i$ and $d_i$ denote the associated sampling times and dosing history. Let $\theta_i$ denote the patient’s individual PK parameters and let $\phi$ denote population and residual-model parameters. The patient-level posterior predictive distribution is $$ p(y_i^{\mathrm{rep}}\mid y_i,t_i,d_i,\mathcal D) = \int p(y_i^{\mathrm{rep}}\mid \theta_i,\phi,t_i,d_i) \,p(\theta_i,\phi\mid y_i,t_i,d_i,\mathcal D)\,d\theta_i\,d\phi , $$ where $\mathcal D$ represents the other information used to estimate the population model. A replicated vector $y_i^{\mathrm{rep}}$ is generated by drawing parameters from the patient’s posterior distribution and then drawing concentrations from the model’s observation distribution at the patient’s recorded times and dosing history.

Constructing a Patient-Level Check

The comparison is meaningful only if the replication reflects the same observation process as the data. For each posterior draw, calculate the model-predicted concentration at each recorded time, then generate a replicated observation according to the residual and assay-error model. If the model includes occasion variability or other observation-level variation, include it as specified. A curve containing only the predicted typical concentrations omits observation variability and is not a replicated data set.

The main assumptions are that the recorded sampling times and dosing history are appropriate for the fit, the PK and observation models define a plausible distribution for concentrations, and the posterior draws adequately represent uncertainty under those models. These checks assess the model as fitted; they cannot correct incorrect data or make an unsuitable model adequate. The introductory material on individual Bayesian forecasts establishes how patient-specific posterior information supports prediction; here the focus is whether replicated observations from that predictive distribution resemble the data used for the individual fit.

A basic graphical check plots the observed concentrations against time and overlays pointwise posterior predictive summaries, such as the median and a central 90% interval at each time. Pointwise means that each interval describes the predictive distribution at one sampling time. It does not mean that 90% of complete replicated concentration–time curves will remain inside all the intervals simultaneously. Connecting the medians can help display the expected time course, but the replicated curves—not just the median line—show the variation in possible observations and their dependence over time.

Fix the patient’s design.
Use the recorded sampling times, dose amounts, dose times, and relevant covariates that were used in the individual fit.

Draw from the posterior.
For each replicate, draw $\theta_i$ and any uncertain model parameters jointly from their posterior distribution. Joint draws preserve posterior dependence among parameters.

Generate observations.
Calculate concentrations under the drawn parameters and generate observations at the recorded times using the fitted observation-error model.

Compare the course and discrepancies.
Plot observed and replicated concentrations over time, then compare any prespecified summary or discrepancy statistic between the observed data and replicates.

Worked Example: Reading a Time-Course Check

Consider a hypothetical patient with four measured concentrations. The table gives the observed values and the median and central 90% pointwise posterior predictive intervals from replicated data at those times.

Time (h)	Observed concentration (mg/L)	Predictive median (mg/L)	Pointwise 90% interval (mg/L)
1	5.2	5.0	4.4 to 5.6
2	4.0	4.0	3.5 to 4.5
4	2.1	2.8	2.2 to 3.4
8	0.9	1.1	0.8 to 1.4

At 1, 2, and 8 hours the observed concentrations lie within the displayed intervals. At 4 hours, $2.1$ mg/L is $0.1$ mg/L below the lower interval limit of $2.2$ mg/L. This is a local mismatch worth inspecting, not proof that the model is wrong: the interval is pointwise, and an occasional observation outside a 90% interval is expected. The full plot can help determine whether the discrepancy is isolated or part of a systematic time-course pattern. Any explanation, such as an inaccurate time record or an unrepresented feature of the concentration profile, requires additional evidence.

Numerical Discrepancy Measures

A plot is often the clearest first check, but a discrepancy measure can summarize a particular feature of fit. The feature should be chosen before interpreting its value: for example, overall standardized deviation from predictive means, maximum deviation at any sample, or a contrast between early and late concentrations. A statistic focused on one feature can miss another. A single overall statistic should therefore complement, not replace, examination of the time-course plot.

One simple choice standardizes each observed concentration by its posterior predictive mean and standard deviation at that time. Let $\mu_{ij}^{\mathrm{pred}}$ and $s_{ij}^{\mathrm{pred}}$ be the predictive mean and standard deviation for the $j$th sample of patient $i$, estimated from posterior predictive draws. Define the sum of squared standardized deviations

T_{\mathrm{obs}} = \sum_{j=1}^{m_i} \left( \frac{y_{ij}-\mu_{ij}^{\mathrm{pred}}} {s_{ij}^{\mathrm{pred}}} \right)^2 .

Here $m_i$ is the number of observed concentrations for the patient. Larger values indicate that the observed vector is farther from its predictive means relative to the predictive spread, according to this particular summary. The calculation is descriptive: its meaning depends on the fitted predictive distributions and the chosen statistic. In particular, it is not automatically a statistic with a standard reference distribution.

Worked Example: Calculating a Standardized Discrepancy

For a hypothetical patient, suppose the observed concentrations are $4.0$, $3.2$, and $2.4$ mg/L. Posterior predictive simulations give means of $3.8$, $3.0$, and $2.2$ mg/L, with predictive standard deviations of $0.4$, $0.8$, and $0.6$ mg/L, respectively. Calculate $T_{\mathrm{obs}}$.

The standardized deviations are $$ \frac{4.0-3.8}{0.4}=\frac{1}{2},\qquad \frac{3.2-3.0}{0.8}=\frac{1}{4},\qquad \frac{2.4-2.2}{0.6}=\frac{1}{3}. $$

Therefore, $$ T_{\mathrm{obs}} = \left(\frac{1}{2}\right)^2+ \left(\frac{1}{4}\right)^2+ \left(\frac{1}{3}\right)^2 = \frac{1}{4}+\frac{1}{16}+\frac{1}{9} = \frac{61}{144} = 0.423611\ldots \approx 0.4236. $$

The value is small in absolute terms, but it has no universal cutoff. Its interpretation requires comparison with the same statistic calculated for replicated data using the same predictive means and standard deviations. Those summaries should be estimated accurately enough that simulation noise does not dominate the comparison.

Comparing the Observed Statistic With Replicates

For a posterior predictive tail proportion, calculate the chosen discrepancy statistic for each replicated data set. The observed value is then compared with the resulting distribution. For a statistic where large values indicate worse fit, the upper-tail proportion is

\widehat{q} = \frac{1}{M}\sum_{r=1}^{M} I\!\left(T(y_i^{\mathrm{rep},r})\geq T(y_i)\right),

where $M$ is the number of replicated data sets and $I(\cdot)$ equals 1 when the condition is true and 0 otherwise. The comparison must use the same definition of $T$ for observed and replicated data. When $T$ uses predictive means and standard deviations estimated from simulations, hold those summaries fixed while calculating the statistics for the observed data and the comparison replicates.

This posterior predictive tail proportion is a descriptive measure of how often the chosen discrepancy is at least as large in replicated data as in the observed data. It is not an ordinary null-hypothesis p-value: the observations help determine the posterior distribution used to generate replicates, so observed and replicated data are not independent under the check. Values near either end can flag a mismatch for the selected feature, but a value near the middle does not prove good fit, and no universal threshold applies to every statistic.

Worked Example: Interpreting a Replicated Tail Proportion

Suppose the observed concentrations for a hypothetical patient are $6$, $3$, and $0$ mg/L. Predictive means at the three times are $5$, $3$, and $1$ mg/L, and each predictive standard deviation is $1$ mg/L. Using the sum of squared standardized deviations gives $$ T_{\mathrm{obs}}=(6-5)^2+(3-3)^2+(0-1)^2=1+0+1=2. $$

Assume predictive summaries were estimated from a large simulation, and eight additional replicated vectors are used for this illustrative comparison. Their values of $T$, calculated with the same fixed means and standard deviations, are $0,1,1,1,2,4,4,4$. Four of the eight values are at least $2$, so $$ \widehat{q}=\frac{4}{8}=0.50. $$

For this statistic and these replicates, the observed discrepancy is not unusual relative to the generated discrepancies. This does not show that every aspect of the patient’s time course is well represented: the statistic may overlook a pattern, and only eight comparison replicates give a coarse estimate of the tail proportion. More simulation reduces Monte Carlo uncertainty, but it does not remove the interpretive limitations of the check.

Diagnostics, Alternatives, and Limitations

When a check looks concerning, first verify that the comparison is being made on the intended scale and at the intended times, and that doses and sampling times were supplied correctly. Check that replicated data include the model’s observation variability; comparing observations only with a smooth predicted concentration curve can make ordinary residual variation look like model failure. Also examine the complete time-course plot: isolated points and repeated departures in one direction or across adjacent times suggest different kinds of mismatch, although neither identifies a cause by itself.

Pointwise intervals have an important limitation. If many samples are checked, some observations may fall outside their pointwise intervals even when the model is reasonable. Conversely, every observation can fall inside its interval while the joint time-course pattern is poorly reproduced. Replicated curves and a statistic targeted at the feature of concern can help assess the joint pattern. A statistic chosen after seeing an unexpected feature is useful for exploration, but its result should be treated as exploratory rather than as a preplanned confirmatory test.

A patient-specific prediction interval across future doses answers a different question: what concentrations might be observed at future times? A posterior predictive check instead compares the observed data used in the individual fit with replicated data at the observed sampling design. Similarly, conditional residual summaries can compare observations with fitted values, but a posterior predictive check propagates posterior parameter uncertainty and observation variability into the replicated data. Neither approach alone establishes that the model is clinically adequate.

Because the patient’s measurements inform the posterior, the predictive distribution can adapt to those measurements. This reuse can make some checks conservative: a model may reproduce an observation partly because that observation helped shape the patient-specific posterior. A reassuring check is therefore not a guarantee of correct individual predictions. Checks should be interpreted alongside the clinical question, the amount and timing of patient data, and the assumptions of the model.

Key takeaway: A patient-level posterior predictive check asks whether the fitted model can generate concentration data resembling this patient’s observed time course under the same dosing and sampling design. Use replicated observations, examine the time pattern, and interpret discrepancy measures as targeted diagnostics—not as definitive tests of model truth.

Common Mistakes

Comparing observations only with a fitted mean curve. The mean curve omits observation variability. Generate replicated concentrations using the fitted observation model.
Treating a pointwise interval as a band for the whole curve. A set of pointwise 90% intervals does not generally contain 90% of complete replicated courses at every time. Examine replicated trajectories for joint behavior.
Calling the tail proportion an ordinary p-value. Posterior predictive replicates are generated from a posterior informed by the observed data. Describe the tail proportion for the chosen discrepancy instead of interpreting it as a null-hypothesis test.
Assuming a reassuring check proves adequate fit. A check only assesses features represented by its plots and statistics. Inspect more than one relevant feature and retain the model’s assumptions in view.
Rounding intermediate values too early. Retain full precision when standardizing and squaring deviations, then round the final statistic for reporting.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What data design should be used to generate patient-level posterior predictive replicates?
A measured concentration lies just outside a pointwise 90% predictive interval. Does this alone establish poor individual fit? Explain.
For one sample, the observed concentration is $5.4$ mg/L, the predictive mean is $5.0$ mg/L, and the predictive standard deviation is $0.5$ mg/L. What is its contribution to the sum of squared standardized deviations?
In 40 replicated data sets, 6 values of an upper-tail discrepancy statistic equal or exceed the observed statistic. Calculate and interpret the tail proportion.
Why can a posterior predictive check look reassuring even if the model has a limitation?

Solutions. (1) Use the patient’s observed sampling times and dosing history, draw parameters from the patient’s posterior, and generate observations using the fitted observation model. (2) No. One point outside a pointwise interval can occur by chance, particularly when several times are checked; inspect the full course and relevant discrepancies. (3) The standardized deviation is $(5.4-5.0)/0.5=0.8$, so its squared contribution is $0.8^2=0.64$. (4) The proportion is $6/40=0.15$. For this discrepancy, 15% of the replicates were at least as extreme in the upper tail; this is descriptive, not an ordinary p-value. (5) The patient’s data inform the posterior used to generate replicates, and a check may assess only selected features. It cannot prove that all model assumptions or predictive uses are adequate.