Predictive Checks for Concentration Tails

Why Examine the Concentration Tails?

A model can describe typical concentrations reasonably well and still miss clinically or operationally important extremes. It may predict high concentrations that are too low, or fail to reproduce the frequency of very low concentrations. These discrepancies can be obscured by summaries focused on the center of a distribution. Tail checks make the question explicit: does the model generate concentrations at the low and high ends in a way that is compatible with what was observed?

A tail check is a comparison between a statistic calculated from the observed data and the same statistic calculated from datasets simulated under the fitted model. The comparison is meaningful only when the simulations reproduce the relevant sampling design and the statistic is defined consistently. For repeated measurements, for example, use one prespecified concentration per subject in a time bin, or simulate complete profiles and calculate the profile-level statistic using the same rule. Treating all measurements as independent when they are clustered within subjects can make a tail look more precisely assessed than it is.

Quantiles and Tail Summaries

For a sample of \(n\) concentrations, the empirical distribution function is the proportion of observations at or below each value. One useful sample quantile convention is its generalized inverse: sort the data as \(x_{(1)}\le\cdots\le x_{(n)}\) and take \(x_{(\lceil nq\rceil)}\), for \(0<q\le1\). This is the nearest-rank convention. Other software may interpolate between observations; use the same convention for the observed data and every simulated replicate, and report it when it could affect the result.

Upper-tail checks often use a high quantile such as \(Q(0.90)\) or \(Q(0.95)\); lower-tail checks may use \(Q(0.10)\) or \(Q(0.05)\). The choice should reflect the question and the amount of data. In a small sample, a very extreme sample quantile can be determined by only one or two observations and will be unstable. A threshold exceedance proportion offers a complementary summary: it directly reports the fraction of concentrations above a concentration of interest, or below a lower threshold.

Comparing Tail Statistics With Simulated Datasets

Let \(T_{\mathrm{obs}}\) be a chosen tail statistic from the observed data. Generate \(B\) replicate datasets under the fitted model, preserving the observation times, subject structure, and any rules for selecting or handling concentrations. Calculate the same statistic in each replicate, giving \(T_1,\ldots,T_B\). A central simulation reference interval is formed from the appropriate lower and upper quantiles of these replicate statistics. For example, the 5th and 95th percentiles give a central 90% interval. If \(T_{\mathrm{obs}}\) falls outside it, the observed statistic is unusual relative to that interval of model-generated datasets.

This interval describes variation in the statistic across replicated datasets under the simulation setup. It includes the variation generated by the model and the finite-sample variation of the statistic at the study’s design. With fitted parameters held fixed, it does not include uncertainty about those parameters. Nor should it be described as a confidence interval for the unknown population quantile: its purpose here is to provide a reference distribution for a sample statistic under the predictive model.

The logic of the check can be justified directly. Under a model that generates datasets like the observed dataset, applying the same statistic and sampling rule to each dataset should produce comparable values. The simulated distribution therefore shows how much a sample tail statistic can vary even when the model is used to generate the data. A discrepancy that is large compared with this variation is more informative than a difference from a single model-predicted quantile.

The rank expression follows because the empirical cumulative probability first reaches or exceeds \(q\) at the first sorted observation whose rank \(k\) satisfies \(k/n\ge q\). The smallest such integer is \(k=\lceil nq\rceil\). This also explains why tied values or discrete concentrations do not require a value with cumulative probability exactly equal to \(q\).

Lower-Tail Checks and Simulated Tail Uncertainty

Lower-tail checks use the same principles, but the direction of a discrepancy needs careful wording. A lower observed \(q\)-quantile than the simulated reference values means that the observed low-end concentrations are lower than the model commonly generates at that quantile. Conversely, a higher observed lower quantile can indicate that the model generates concentrations that are too low in that part of the distribution. A threshold check below a specified concentration can make the practical interpretation clearer, provided the threshold and handling of measurements below quantification are defined consistently.

The simulated reference interval also helps distinguish an apparent difference from a stable tail discrepancy. Tail statistics are often variable: with few subjects, a single unusually high value can move an upper quantile sharply. The replicate distribution makes this finite-sample instability visible under the assumed model and design. It does not remove uncertainty or guarantee that the simulated distribution is correct.

Assumptions, Diagnostics, and Alternatives

A defensible tail check depends on several conditions. The summary and any threshold should be chosen to address a defined question, rather than selected only after viewing results. Simulated datasets should represent the same design as the observed data, including the number of subjects, observation times, relevant strata, and dependence within subjects. Apply identical data-processing and quantile conventions to observed and simulated concentrations. If observations below quantification are present, the censoring and reporting rule must also be applied consistently; substituting an arbitrary value can distort the lower tail.

Check for warning signs before interpreting a discrepancy. A tail quantile may be unstable when only a few observations define it; ties can make threshold proportions jump in coarse increments; and a small number of simulated datasets can make the reference limits noisy. Increase the number of simulations if the limits or rank estimates change materially across runs. Inspect whether the comparison is being driven by one subject, a particular time window, or a subgroup. Repeated observations should not be counted as independent subjects unless the statistic and simulation explicitly account for that structure.

Tail quantiles, threshold proportions, and predictive-interval coverage answer related but different questions. A quantile compares a location in the concentration distribution; a threshold proportion counts observations on one side of a specified value; interval coverage asks how often observations fall within a predictive interval. A visual predictive check can display quantile behavior across time or groups, while normalized prediction distribution errors summarize discrepancies on a transformed scale. These alternatives can reveal patterns that one tail statistic misses, but none substitutes for choosing a summary matched to the question. A tail check focused on one endpoint or stratum also does not establish that multiple endpoints or correlated profiles are jointly well predicted.

Finally, a good tail check is conditional on the predictive simulation being appropriate. When model parameters are held at their fitted estimates, the replicate distribution reflects model-generated and finite-sample variation conditional on those estimates. It does not account for parameter uncertainty. An observed statistic inside a reference interval is therefore not proof that the model is correct, just as one outside it is not proof of a unique failure mechanism.