Choose a Statistic to Match the Predictive Question
A predictive check compares a feature of the observed data with the same feature calculated from data replicated under a fitted population pharmacokinetic model. The statistic is not a decorative summary: it determines which discrepancies the check can reveal. A model can reproduce average concentrations while missing variability in the upper tail, or reproduce a peak while misrepresenting total exposure. The right statistic therefore depends on the specific feature that matters.
For example, concentration quantiles address the distribution of concentrations at selected times; a proportion above a threshold addresses the frequency of high or low values; a maximum addresses peak levels; and an area-under-the-curve summary addresses integrated exposure. None is a universal measure of model adequacy. A useful check begins by stating the feature of the data the model is expected to reproduce.
Let \(D_{\mathrm{obs}}\) denote the observed data and \(D^{(b)}\) the \(b\)th replicated data set, generated under the predictive-check procedure. The basic comparison is between \(S(D_{\mathrm{obs}})\) and the collection \(S(D^{(1)}),\ldots,S(D^{(B)})\). The replicate data must reflect the intended target: for example, population-level or conditional prediction, the observed dosing and covariates, and the relevant sampling design. The earlier material on predictive-check targets and replicate-data generation establishes those choices. A statistic cannot repair a mismatch between the target data and the data being simulated.
Statistics for Concentration Distributions
When the question concerns the distribution of concentrations at a particular time, quantiles are often more informative than a mean alone. A median can reveal a shift in a typical concentration; lower and upper quantiles can reveal discrepancies in spread or asymmetry. Calculate these summaries at specified times using the observed concentrations available for the target and the corresponding simulated observations. If several times matter, retain the time-specific summaries rather than combining them into one number that might hide a local discrepancy.
A quantile is a location in a distribution: for example, the \(p\) quantile is a value at or below which approximately a proportion \(p\) of observations falls. In a finite sample, an empirical quantile requires a convention. One simple convention used in the examples below is the nearest-rank definition: sort the \(n\) values and choose the value at rank \(\lceil pn\rceil\), for \(0<p\leq1\). Other standard software conventions interpolate between order statistics. For an actual analysis, state and apply one convention consistently to observed and replicated data.
Worked Example: Comparing Concentration Quantiles at One Time
Suppose a hypothetical check concerns concentrations at 2 hours after dosing. The eight observed values, in mg/L, are \(1, 2, 2, 3, 4, 5, 8,\) and \(12\). Use the nearest-rank definition to calculate the median and the 90th percentile. Then compare them with summaries from 500 simulated data sets, whose median values across simulations are 3.5 mg/L for the median statistic and 9.0 mg/L for the 90th-percentile statistic.
For the median, \(p=0.5\), so the selected rank is \(\lceil 0.5(8)\rceil=4\). The fourth sorted observation is 3 mg/L. For the 90th percentile, \(p=0.9\), so the rank is \(\lceil 0.9(8)\rceil=8\); the eighth value is 12 mg/L.
Thus, the observed median is 3 mg/L, compared with a simulated median-statistic value of 3.5 mg/L, while the observed 90th percentile is 12 mg/L, compared with 9.0 mg/L. The difference is concentrated in the upper tail rather than the typical concentration. This comparison motivates inspecting the replicate distributions for both statistics; it does not, by itself, establish why the tail differs or whether the model is adequate in every other respect.
Quantiles should be chosen to represent a meaningful part of the distribution. A central quantile is generally more stable than an extreme quantile in a small data set, because the latter depends on very few observations. Quantiles can also be sensitive to how observations are pooled. If subjects contribute different numbers of samples, a quantile across all records gives more weight to subjects with more observations. If the target is a typical subject, a subject-level summary followed by a population summary may better match the question.
Thresholds, Peaks, and Exposure
A proportion above a threshold is a direct statistic when the concern is how often concentrations exceed a clinically or scientifically meaningful level. For a threshold \(c\), define an indicator \(I(C>c)\), equal to 1 when the condition holds and 0 otherwise. The observed exceedance proportion is the mean of these indicators. Specify whether the unit is a concentration observation, a subject, or a subject-time combination: these answer different questions.
Worked Example: Proportion Above a Concentration Threshold
In a hypothetical data set, 20 concentration observations at a prespecified time are available, and 7 exceed 4 mg/L. The observed proportion above the threshold is
Thus, 35% of these observations exceed 4 mg/L. Suppose the same calculation in each of 500 simulated data sets gives a median proportion of 0.20, with simulated proportions ranging from 0.05 to 0.40. The observed value of 0.35 is toward the high end of these simulated values, suggesting that this check should focus on the model's reproduction of threshold exceedance. The comparison is about observation-level exceedance at that time; it does not imply that 35% of subjects ever exceeded the threshold.
If the target instead is the fraction of subjects who ever exceed 4 mg/L, calculate one indicator per subject: it is 1 if that subject has at least one qualifying concentration and 0 otherwise. Counting all records would answer a different question and could let subjects with more samples contribute more heavily.
A short justification clarifies why repeated observations require care. If \(n\) indicators \(I_j\) are independent and each has probability \(p\) of equaling 1, then \(E(I_j)=p\) and \(\mathrm{Var}(I_j)=p(1-p)\). For \(\hat{p}=n^{-1}\sum_j I_j\),
The variance calculation uses independence to make the covariances zero. Concentrations from the same person are often correlated, so treating every record as an independent unit can make a proportion seem more precise than it is and can obscure the intended population quantity. In a predictive check, the replicated data should preserve the relevant repeated-measures structure, and the statistic should use the same subject or observation unit as the stated target.
Peak concentration and exposure summaries target different profile features. A sampled maximum, \(C_{\max,\mathrm{sampled}}\), is the largest concentration among the recorded sampling times. It is not necessarily the true maximum of the underlying continuous concentration-time curve: if the actual peak occurs between samples, the sampled maximum may be lower. Area under the concentration-time curve (AUC) summarizes exposure over a specified interval. With observations at times \(t_1,\ldots,t_m\), the linear trapezoidal approximation is
Worked Example: Sampled Peak and Trapezoidal Exposure
Consider a hypothetical concentration profile with values \(0, 8, 6,\) and \(2\) mg/L at times \(0, 2, 4,\) and \(8\) hours, respectively. The largest recorded concentration is \(8\) mg/L, so the sampled peak is 8 mg/L. This identifies the highest observed value, not necessarily the maximum between the sampled times.
The trapezoidal AUC from 0 to 8 hours is
Suppose, hypothetically, the observed profile has this AUC and sampled peak, while the medians of the corresponding statistics across simulated profiles are 34 mg·h/L and 7.2 mg/L. The AUC comparison targets total exposure over the specified interval; the peak comparison targets the largest recorded concentration. They should be considered separately because similar AUC values can arise from profiles with different peaks and shapes.
Practical Selection and Failure Modes
A defensible selection process starts with the scientific question and makes the data unit explicit. State whether the aim concerns the distribution at a time, the frequency of crossing a threshold, an individual profile's peak, or exposure over an interval. Then define the calculation precisely, including time window, threshold, quantile convention, treatment of missing or unusable records, and whether summaries are observation-level or subject-level. Apply the identical rules to observed and replicated data.
Translate the concern into a target such as upper-tail concentration at 6 hours, fraction of subjects ever above a threshold, sampled peak, or AUC over a stated interval.
Write down exactly which observations enter, how records are grouped, and how the summary is calculated. Check that this definition corresponds to the scientific question.
Calculate the same statistic with the same sampling schedule and processing rules. A statistic computed from latent continuous curves is not directly comparable to one computed from sparse observed samples.
A discrepancy points to a feature the model may not reproduce. Check data handling, design alignment, and whether other statistics tell a consistent story before attributing it to a particular model component.
Several common alternatives answer related but non-identical questions. A mean concentration summarizes the center on the original scale but can conceal tail behavior. Quantiles describe selected locations in the distribution but do not directly report the frequency above a clinically chosen cutoff. A threshold proportion is easy to interpret but depends on the chosen cutoff and discards how far above or below it values lie. A sampled maximum is sensitive to sample size, sampling times, and measurement error. AUC is an integrated measure and can conceal differences in curve shape. A time-specific statistic preserves when a feature occurs, whereas a single summary across time may blur that information.
| Statistic | Best aligned question | Important limitation |
|---|---|---|
| Concentration quantile at a specified time | Does the model reproduce a selected part of the concentration distribution then? | Tail quantiles can be unstable with limited data; pooling choices affect weighting. |
| Proportion above a threshold | How often do observations or subjects cross a meaningful cutoff? | The threshold and unit of analysis must be specified; distance beyond the cutoff is lost. |
| Sampled maximum concentration | Does the model reproduce the highest recorded concentration? | Depends on sampling times and number of observations; it may miss the true peak. |
| AUC over a stated interval | Does the model reproduce integrated exposure over that interval? | Different concentration-time shapes can have similar AUC. |
Diagnostics for a poorly chosen statistic include a summary that is almost unchanged by the suspected discrepancy, an extreme quantile driven by one or two records, a threshold proportion dominated by subjects with unusually dense sampling, or an AUC calculated over different time windows in observed and replicated data. If the statistic behaves implausibly, verify its definition and input records before drawing a model conclusion. A predictive check is also limited by the model and predictive procedure used to generate replicates: agreement for one statistic does not establish agreement for unexamined features.
Selecting several statistics can provide complementary views, but adding summaries without a clear question can make interpretation diffuse. Prioritize features tied to the intended use of the model, and report the definitions and units so that the comparison is reproducible. The key discipline is alignment: the statistic, data unit, observation schedule, and scientific question should all refer to the same feature.
Check Your Understanding
Answer each question, then compare your reasoning with the concise solutions below.
- A study question asks whether the model captures the upper tail of concentrations at 6 hours. Which statistic is a direct choice, and what must be held consistent?
- Of 25 observations, 5 exceed 3 mg/L. Calculate the observation-level exceedance proportion.
- Why is the largest recorded concentration not automatically the true peak concentration?
- Two profiles have similar AUC values but one has a sharper peak. Which additional statistic would help distinguish them?
- Why can a pooled observation-level proportion answer a different question from the fraction of subjects who ever exceed a threshold?
Solutions. (1) Use an upper quantile, such as the 90th percentile, at 6 hours; use the same quantile convention, time definition, and sampling basis for observed and replicated data. (2) \(5/25=0.20\), or 20%. (3) The maximum may occur between scheduled measurements, so the largest sampled value can be lower than the underlying profile maximum. (4) Compare a peak statistic, such as the sampled maximum when the sampling schedule is the target. (5) The pooled proportion counts concentration records, so subjects with more records can contribute more; the subject-level statistic counts each subject once based on whether any qualifying exceedance occurred.