Why Binning and Summarizing Matter
A prediction-corrected visual predictive check (pcVPC) compares observed outcomes with replicate outcomes after adjusting for differences in typical predictions across records. That adjustment makes outcomes more comparable on a common scale; it does not remove the need to summarize them over time. The choice of time bins determines which observations are grouped together, while the quantiles within each bin describe the central tendency and spread being checked.
The central task is to make the observed and simulated summaries comparable. Use the same time bins, the same prediction-correction rule, and the same quantile definition for the observed data and every simulated replicate. A bin that is too narrow may contain too few observations for stable quantiles; a bin that is too wide may average over a meaningful change in the outcome over time.
The reference prediction and the prediction-correction idea are established in the earlier material on choosing a reference prediction and calculating prediction-corrected observations. Here, the focus is how to group records and calculate the resulting summaries. The ratio \(P_{\mathrm{ref},b}/P_{ij}\) is dimensionless, so a corrected outcome retains the units of \(Y_{ij}\), such as mg/L.
Choosing Time Bins
Time bins should be selected using the observation schedule and the time resolution needed to reveal model misfit. Boundaries may be fixed in advance at scientifically meaningful times, or chosen after inspecting the distribution of observation times. In either case, record the boundaries precisely and apply them identically to observed and simulated data. A practical aim is to have enough observed records in each bin to estimate the chosen quantiles without making bins so broad that they conceal time-dependent structure.
There is no universally correct minimum count per bin. Quantiles in small samples are intrinsically coarse: with five observations, for example, an empirical 10th percentile may be the smallest observed value. Simulations show how much variability in a bin summary is expected under the fitted model, but they cannot supply information about time regions with no observed data. Bins should therefore be assessed for both count and temporal width, not count alone.
Worked Example: Selecting Bins for Uneven Sampling
Suppose a hypothetical study has observation times, in hours, of 1.0, 1.2, 1.5, 1.8, 2.2, 2.7, 3.1, 3.5, 4.4, 5.2, 6.3, 8.0, 9.0, 10.5, and 12.0. The goal is to summarize early, middle, and later observations without leaving a bin nearly empty.
Consider the boundaries \([0,3)\), \([3,7)\), and \([7,13]\) hours. The first bin contains 1.0, 1.2, 1.5, 1.8, 2.2, and 2.7: \(n_1=6\). The second contains 3.1, 3.5, 4.4, 5.2, and 6.3: \(n_2=5\). The third contains 8.0, 9.0, 10.5, and 12.0: \(n_3=4\). These counts provide a transparent starting point for three summaries, though the last bin’s quantiles will be particularly coarse.
If the scientific question requires finer resolution around 3 hours, moving the first boundary to 2 hours would put only four observations in the first bin and two in the next interval from 2 to 3 hours. That split may be too sparse for stable lower and upper quantiles. A defensible choice is therefore to keep the three bins, or to use a pre-specified boundary only if the resulting uncertainty is acknowledged. The bin scheme should not be adjusted simply to make observed quantiles look closer to the simulations.
Calculating Corrected Quantiles
Once the bins are set, apply the correction record by record before calculating the observed quantiles. For a record in bin \(b\), its observed value is multiplied by the ratio of that bin’s reference prediction to the record’s typical prediction. Then sort the corrected values in the bin and calculate the selected empirical quantiles. State the quantile convention, particularly when bins are small, because software packages may use different interpolation rules.
For a simple and reproducible convention, this tutorial uses the empirical inverse-CDF quantile: for \(n\) sorted values \(x_{(1)}\leq\cdots\leq x_{(n)}\), \(Q_p=x_{(\lceil np\rceil)}\), with the index restricted to 1 through \(n\). This convention does not interpolate between observed values. Other standard empirical quantile conventions are also used; the essential requirement for a pcVPC is to use the same convention for observed values and every replicate.
Worked Example: Correcting Observations and Finding the Median
In a hypothetical time bin, suppose the selected reference prediction is \(P_{\mathrm{ref},b}=10\) mg/L. Three observed records have typical predictions of 8, 10, and 12 mg/L, with observed outcomes of 7.2, 11.0, and 9.6 mg/L, respectively.
Apply the correction to each record. The corrected values are \(7.2(10/8)=9.0\) mg/L, \(11.0(10/10)=11.0\) mg/L, and \(9.6(10/12)=8.0\) mg/L. Thus, the corrected values in sorted order are 8.0, 9.0, and 11.0 mg/L.
For \(n=3\), the empirical inverse-CDF median is \(Q_{0.5}=x_{(\lceil3(0.5)\rceil)}=x_{(2)}=9.0\) mg/L. The uncorrected outcomes have median 9.6 mg/L, but that is not the pcVPC summary: the target summary is calculated from the corrected values.
Simulated Quantiles and Prediction Intervals
Let \(K\) denote the number of simulated replicates. For each replicate \(k\), calculate the simulated outcome at each relevant record, apply the same correction multiplier used for that record, and calculate the selected quantile within each time bin. This yields \(K\) replicate quantiles for each bin and each quantile level. The prediction interval is obtained from the distribution of these replicate quantiles.
For example, a central 90% pointwise prediction interval for the median in bin \(b\) is formed from the 5th and 95th percentiles of the \(K\) simulated medians. It describes the range that contains the central 90% of the simulated bin medians under the chosen simulation setup and fitted model, subject to the empirical quantile method used. It is not a confidence interval for a model parameter, nor does it mean there is a 90% probability that the true median lies within the interval.
A pointwise interval is calculated separately for each bin and quantile. It does not automatically provide simultaneous coverage across all bins, all quantiles, or the entire plotted curve. The interval also depends on what the simulations represent: simulation of new subjects and observations assesses a different predictive target from simulation conditional on fixed subject-level information. The simulation design must match the target of the predictive check.
Worked Example: Comparing an Observed Median with Its Prediction Interval
Continue the hypothetical bin from the preceding example, where the observed corrected median is 9.0 mg/L. Suppose 100 simulated replicates produce 100 corrected medians in that bin. After sorting them, the 5th order statistic is 8.1 mg/L and the 95th order statistic is 10.4 mg/L. Using the empirical inverse-CDF convention, the central 90% interval uses ranks \(\lceil100(0.05)\rceil=5\) and \(\lceil100(0.95)\rceil=95\), giving \([8.1,10.4]\) mg/L.
The observed median of 9.0 mg/L lies within this pointwise interval because \(8.1\leq9.0\leq10.4\). In this bin, the observed median is therefore not unusually low or high relative to the central 90% of simulated medians under this construction. That is a descriptive comparison, not proof that the model is correct. Other bins or quantiles may show discrepancies, and a pointwise interval does not account for searching across many plotted summaries.
If the observed median had been 11.2 mg/L, it would exceed the upper limit by \(11.2-10.4=0.8\) mg/L. The appropriate description would be that the observed corrected median is above the central 90% pointwise prediction interval in this bin. It would not be correct to interpret this as a formal test with a guaranteed 10% error rate across the whole pcVPC.
Justification for the Correction and the Summary
The correction makes outcomes comparable when their typical predictions differ, provided the multiplicative scale is appropriate. To see the basic justification, suppose outcomes in a bin can be represented as \(Y_{ij}=P_{ij}\varepsilon_{ij}\), where \(\varepsilon_{ij}\) is a multiplicative deviation whose distribution does not systematically change across records after accounting for the predictive target. Then
The record-specific typical prediction cancels, putting the corrected value on the reference-prediction scale. If the multiplicative deviations have a common distribution, their corrected quantiles are comparable within that bin. This is a scale argument, not a guarantee that correction makes all records exchangeable or removes all sources of heterogeneity.
In particular, a proportional correction should not be treated as automatically appropriate for additive error. If outcomes follow an additive structure, multiplying by a prediction ratio also rescales the additive residual, potentially changing its spread across records. Prediction correction is most defensible when the scale and error structure support the chosen transformation, and interpretation should remain tied to that corrected scale.
Diagnostics, Alternatives, and Limitations
Review bin counts and time spans alongside the plotted summaries. A steep change in the reference prediction within a wide bin, a sudden jump in the observed quantiles at a boundary, or highly variable simulated quantiles can indicate that the binning is obscuring structure or that data are too sparse for stable summaries. Also check that observed and simulated values use the same time assignments, correction factors, outcome units, and quantile convention. A mismatch in any of these can create an apparent discrepancy that is computational rather than model-based.
A useful alternative is to use narrower bins where observations are dense and wider bins where they are sparse, if the resulting boundaries still preserve the time resolution needed for the scientific question. Fixed-width bins are easy to communicate, while bins with approximately balanced counts can stabilize quantile estimates. Neither rule is universally superior: fixed-width bins may be empty or sparse, and equal-count bins may span very different durations. The number of bins should be chosen with both interpretability and precision in mind.
Binning also has a basic limitation: summaries discard information within each bin. Two very different time patterns can have the same bin quantile, and changing a boundary can change a summary even when the underlying data do not change. For this reason, retain the bin counts, boundaries, and quantile definitions when reporting results. If a bin contains very few observations, describe its summary as imprecise rather than treating the simulations as a substitute for observed information.
Worked Example: Deciding Whether to Merge a Sparse Bin
Suppose a hypothetical initial scheme has three bins with 8, 2, and 11 observed records. The middle bin covers 2 to 3 hours, and the neighboring bins cover 0 to 2 and 3 to 8 hours. With only two records in the middle bin, the empirical 10th and 90th percentiles are the minimum and maximum, respectively: for \(n=2\), \(\lceil2(0.10)\rceil=1\) and \(\lceil2(0.90)\rceil=2\). These quantiles provide almost no resolution about the distribution within that bin.
If the scientific question does not require a distinct summary from 2 to 3 hours, merging the middle and later bins gives \(2+11=13\) observations across 2 to 8 hours. The 10th percentile then uses rank \(\lceil13(0.10)\rceil=2\), and the 90th uses rank \(\lceil13(0.90)\rceil=12\). This gives more order-statistic resolution, but the merged bin spans six hours and may conceal a time trend. The decision is therefore conditional: merge only if that loss of temporal resolution is acceptable; otherwise retain the narrow bin and report its quantiles as highly coarse, or revise the scheme using a scientifically defensible boundary.
Choose the quantiles of interest, the empirical quantile convention, and the simulation target.
Use explicit boundaries, count observed records in every bin, and check that bin widths preserve meaningful time resolution.
Apply the selected reference-prediction multiplier to observed and replicate outcomes at corresponding records.
Calculate observed quantiles, calculate the same quantiles for each replicate, and form pointwise prediction intervals from the replicate quantiles.
Relate discrepancies to bin counts, temporal resolution, and the predictive target; do not treat pointwise agreement as global validation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why must the same time bins and correction rule be used for observed outcomes and simulated replicates?
- A bin contains 10 corrected observations. Under the empirical inverse-CDF convention used here, which ordered observation is the 50th percentile?
- What does a central 90% pointwise prediction interval for a bin median describe, and what does it not describe?
- A bin has one observed record. What are the empirical 10th, 50th, and 90th percentiles under the stated convention, and what limitation should be reported?
Solutions. (1) Otherwise, differences between observed and simulated summaries could arise from different grouping or scaling rather than from the predictive comparison. (2) The rank is \(\lceil10(0.50)\rceil=5\), so it is the fifth ordered observation. (3) It describes the central 90% of simulated bin medians under the specified simulation setup; it is not a confidence interval for a parameter, a probability statement about the true median, or simultaneous coverage of the full plot. (4) All three empirical quantiles are the sole observed value, because every percentile selects rank \(\lceil1p\rceil=1\). This does not imply that the bin’s underlying distribution has no spread; it means the observed sample is too small to estimate that spread usefully.