What a Reference Envelope Represents
A predictive check compares a statistic calculated from the observed data with the same statistic calculated from replicate datasets generated under a fitted population pharmacokinetic model. A reference envelope summarizes the range of replicate statistics expected under that model. It gives a visual scale for judging whether the observed statistic is unusual relative to model-based replication; it is not, by itself, a confidence interval for a model parameter.
The envelope is meaningful only after the predictive-check target and statistic have been specified. The earlier material on selecting statistics for predictive checks addresses that choice. Here, suppose the statistic has been fixed and is evaluated at one or more coordinates, such as time points or prespecified groups. Denote the observed statistic at coordinate \(j\) by \(T^{\mathrm{obs}}_j\), and its value in replicate dataset \(b\) by \(T^{(b)}_j\), for \(b=1,\ldots,B\). The coordinate might be a single summary, or one location on a curve of summaries.
A typical pointwise 95% envelope uses the 2.5th and 97.5th percentiles of the replicate statistics at each coordinate. If \(L_j\) and \(U_j\) denote these bounds, the observed statistic is inside at coordinate \(j\) when \(L_j \leq T^{\mathrm{obs}}_j \leq U_j\). The nominal coverage refers to the model-based distribution at that one coordinate, under the assumptions used to generate replicates. It does not mean there is a 95% probability that the true model is correct, nor that a parameter lies in the band.
How the Bounds Are Constructed
The construction begins with a clear simulation target. Replicate datasets should be generated under the fitted model using the design features relevant to that target, and the same statistic should be calculated from each replicate as from the observed data. For example, if the statistic is a residual summary by a prespecified coordinate, use the same definition and calculation for each replicate. A mismatch between the observed and simulated calculations makes the comparison difficult to interpret.
In a common plug-in check, the fitted population and individual-level model parameters are held fixed while new data are simulated from the model. This asks what the statistic would look like if the fitted model generated new data under the specified design. It does not automatically include uncertainty in the estimated parameters. Whether to condition on fixed estimates or incorporate parameter uncertainty depends on the diagnostic question; the choice should be stated because it can change the width of the reference distribution.
For a pointwise central envelope at nominal level \(1-\alpha\), use the \(\alpha/2\) and \(1-\alpha/2\) quantiles of the replicate statistics at each coordinate:
Here \(Q_p\) is the empirical \(p\) quantile, calculated using a stated or software-standard quantile convention. Other quantile choices are possible. For example, a one-sided check may use a lower or upper tail bound, while a central interval is suited to detecting departures in either direction. The choice should match the discrepancy of interest rather than being selected after looking at the observed curve.
Worked Example: Reading a Pointwise Envelope
Suppose 1,000 hypothetical replicate datasets are generated from a fitted PK model. At one prespecified coordinate, the statistic is the median concentration, in mg/L. The empirical 2.5th and 97.5th percentiles of the 1,000 simulated medians are 8.9 mg/L and 11.2 mg/L. The observed median is 10.8 mg/L.
The pointwise envelope is \([8.9,11.2]\) mg/L. Comparing the observed value with the bounds gives \(8.9 \leq 10.8 \leq 11.2\), so the observed median is inside this pointwise 95% envelope. Its location within the envelope is not evidence that the model is correct in every respect; it says only that this selected summary at this coordinate is not outside these marginal bounds.
If instead the observed median were 11.6 mg/L, then \(11.6 > 11.2\) mg/L, an upper-tail excursion at this coordinate. That is a signal to investigate the discrepancy in context, not an automatic formal rejection of the model.
Pointwise Coverage Is Not Coverage of the Whole Curve
A pointwise envelope is calibrated one coordinate at a time. Even when each coordinate has 95% marginal coverage under the model, the probability that all coordinates lie inside their pointwise bounds is generally less than 95%. The chance of at least one excursion rises as more coordinates are examined, although its exact value depends on the dependence among the summaries.
For illustration, if 20 coordinates were independent and each had a 5% probability of falling outside its central pointwise envelope, the probability of at least one excursion would be \(1-(1-0.05)^{20}=1-0.95^{20}\approx 0.6415\). Thus, even under that simplifying independence assumption, a curve with one or more pointwise excursions would occur about 64.15% of the time. In practice, summaries at nearby coordinates are often dependent, so this independence calculation is not a general calibration rule.
Worked Example: Interpreting Several Pointwise Excursions
Consider a hypothetical check with 20 coordinates. Suppose three observed summaries lie outside their respective pointwise 95% bounds: two above the upper bounds and one below a lower bound. Each individual excursion is visible, but the pointwise envelope does not tell us the probability of seeing three or more excursions across this particular set of dependent coordinates.
Under the hypothetical assumption of independence, the expected number of coordinates outside their bounds would be \(20 \times 0.05=1\). The observed count of three is larger than that expectation, but an expectation is not a calibrated test statistic: it does not give the probability of this pattern or account for dependence. The appropriate reading is that the pattern deserves investigation, especially if the excursions are coherent in direction or occur in a clinically relevant region. A simultaneous reference envelope can instead assess whether the overall collection of summaries departs unusually far from its simulated reference.
Constructing and Interpreting a Simultaneous Envelope
A simultaneous envelope is designed to control coverage across a collection of coordinates together. One practical approach standardizes each replicate statistic relative to the replicate distribution at that coordinate, then calibrates the largest absolute standardized deviation across coordinates. Let \(m_j\) be a center of the simulated statistics at coordinate \(j\), such as their mean or median, and let \(s_j\) be a positive scale measure, such as their standard deviation. For replicate \(b\), calculate \(M_b=\max_j |(T^{(b)}_j-m_j)/s_j|\). Choose \(c\) as the \(1-\alpha\) quantile of the replicate maxima. A corresponding simultaneous envelope is \(m_j \pm c s_j\) at each coordinate.
The justification follows from the definition of the maximum. For any vector of statistics, all its coordinates are inside this band exactly when every standardized absolute deviation is no larger than \(c\), which is equivalent to the maximum being no larger than \(c\):
Therefore, if \(c\) is the \(1-\alpha\) quantile of the maximum under the model, the replicate vectors fall entirely within the band with probability approximately \(1-\alpha\), subject to simulation and quantile-estimation error. Applying this calibration to the observed vector asks whether its largest standardized departure exceeds the model-based cutoff. It is the collection of departures, rather than each coordinate considered in isolation, that is being assessed.
This construction depends on the chosen center, scale, coordinates, and statistic. A zero or nearly zero \(s_j\) makes standardization undefined or unstable; that coordinate needs a suitable alternative scale or separate treatment. Other simultaneous-band methods are possible, including calibrating a maximum one-sided deviation when only one direction matters. The band should be described precisely enough that its target and calibration are clear.
Worked Example: Checking a Simultaneous Envelope
Suppose three hypothetical coordinates have simulated centers \(m=(10,7,4)\) mg/L and scales \(s=(1,0.5,2)\) mg/L. The 95th percentile of the replicate maximum absolute standardized deviation is \(c=2.4\). The simultaneous bounds are calculated coordinate by coordinate.
At coordinate 1, the bounds are \(10 \pm 2.4(1)\), or \([7.6,12.4]\) mg/L. At coordinate 2, they are \(7 \pm 2.4(0.5)=[5.8,8.2]\) mg/L. At coordinate 3, they are \(4 \pm 2.4(2)=[-0.8,8.8]\) mg/L. These values are illustrative; a negative lower bound may be mathematically possible for a standardized construction even when concentrations themselves cannot be negative.
Let the observed statistics be \((9,7.25,8.2)\) mg/L. Their standardized deviations are \((9-10)/1=-1\), \((7.25-7)/0.5=0.5\), and \((8.2-4)/2=2.1\). The maximum absolute deviation is \(2.1\), which is below \(2.4\), so the observed vector is inside the simultaneous envelope. If the third observed value were 9.6 mg/L instead, its standardized deviation would be \((9.6-4)/2=2.8\); the maximum would exceed 2.4, and the vector would fall outside the band.
This is a model-based, joint comparison across these three coordinates. It does not identify which structural or stochastic feature caused an excursion, and it does not convert the plotted band into a probability statement about the fitted parameters.
Assumptions, Diagnostics, and Failure Modes
Envelope interpretation relies on several practical conditions. First, the simulated replicates must represent the predictive target: the model, design, and calculation of the statistic should match the question being asked. Second, the statistic and coordinates should be defined consistently for observed and simulated data. Third, the selected reference distribution must be a reasonable representation of variation under the model. A plug-in envelope conditions on fitted parameter estimates; it does not generally account for estimation uncertainty unless that uncertainty is explicitly incorporated. Finally, the number of replicates must be adequate for estimating the desired quantiles and, for simultaneous bands, the tail of the maximum distribution. Monte Carlo error can make estimated bounds unstable, particularly for extreme quantiles.
Useful diagnostics include checking that each replicate produced the intended statistic, inspecting the simulated distributions for skewness or outliers, and confirming that the envelope width is not dominated by a few unusual replicates. Compare the observed pattern with both the envelope and the simulated center: a value can be inside a wide band yet consistently displaced from the center. Also inspect whether excursions are isolated, directional, or sustained across neighboring coordinates. These patterns guide follow-up but do not identify a cause on their own.
A conspicuously narrow envelope may reflect too little simulated variability, an incorrect simulation design, or an unstable scale estimate. A very wide envelope can obscure meaningful discrepancies. If many summaries or coordinates are inspected, the chance of at least one pointwise excursion increases; reporting only the most striking point without noting this multiplicity can exaggerate its evidential weight. Conversely, an observed curve inside a simultaneous band does not prove adequate fit: the chosen summaries may be insensitive to the model feature that is wrong, and a finite check has limited ability to detect some departures.
There is also a dependence issue when parameters are estimated from the observed dataset and then held fixed for simulation. The observed statistic and replicate statistics need not behave like fully independent draws from a known reference distribution, because the observed data informed the fitted parameters. A plug-in predictive check remains useful as a diagnostic, but its nominal envelope should not automatically be treated as an exact frequentist test. If the target is the behavior of an entire fitting procedure, a simulation design that repeats the fitting step may be more appropriate; that is a different target from checking new data conditional on a fixed fitted model.
Choosing the Right Interpretation
| Envelope | What its nominal coverage concerns | Appropriate reading of an excursion |
|---|---|---|
| Pointwise | One coordinate at a time | The observed summary is unusual at that coordinate relative to its marginal simulated distribution. |
| Simultaneous | The whole prespecified set of coordinates together | The observed vector has a joint departure larger than the calibrated model-based threshold. |
| Parameter confidence interval | A parameter under an estimation procedure | Not the same object as a predictive-check envelope. |
The main alternative to a pointwise envelope is not simply “a better interval,” but a different calibration for a different question. Pointwise bounds are easy to read and can localize departures, but do not control the probability of any excursion across a curve. Simultaneous bounds address an overall curve-level question, but can be wider and less sensitive to a localized departure. A useful report states which envelope was used, its nominal level, how it was constructed, how parameters were treated during simulation, and what statistic and coordinates it covers.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A pointwise 90% envelope is built for a statistic at one coordinate. What does its nominal level describe, and what does it not describe?
- At 30 independent coordinates, each with a 5% pointwise excursion probability, calculate the probability of at least one excursion. Why should this result not automatically be applied to a real curve?
- A simultaneous band is calibrated using the 95th percentile of the maximum absolute standardized deviation. What event does the band cover at approximately the 95% level?
- Give two reasons an observed statistic might fall outside a pointwise envelope, and explain why the excursion alone does not identify the cause.
- Why can an observed curve inside a simultaneous envelope not establish that the model is adequate in every respect?
Solutions. (1) It describes marginal coverage at that one coordinate under the model-based reference distribution and stated simulation assumptions. It does not describe parameter uncertainty or guarantee 90% coverage of an entire curve. (2) Under independence, the probability is \(1-0.95^{30}\approx 0.7854\), or about 78.54%. Real curve summaries are often dependent, so the joint probability can differ. (3) The event is that every coordinate’s absolute standardized deviation is no larger than the calibrated cutoff; equivalently, the largest such deviation is no larger than that cutoff. (4) The model may fail to reproduce the relevant feature, or the excursion may arise from ordinary simulation variation; mismatched calculations or an unsuitable simulation design are additional possibilities. The comparison flags a discrepancy but does not diagnose its source. (5) The selected statistic may not be sensitive to other forms of misspecification, and an envelope assesses only the chosen summaries, coordinates, and simulation target.