Why Irregular Sampling Matters in a VPC
In a visual predictive check (VPC), observed data summaries are compared with summaries calculated from replicate data generated under the fitted model. When sampling times are irregular, some regions of time may contain many observations and others very few. In addition, the observations within a time bin may come from different protocols, each contributing a different share of the data. These features affect what a pooled VPC summary represents and how precisely it is estimated.
Uneven sampling density is not, by itself, evidence of model failure. If the model is evaluated at the same observation times and under the same protocol composition as the observed data, the simulations reflect that design. The central questions are instead: what population of observations does each bin summarize, and does the simulated comparison use that same population?
A VPC bin does not necessarily estimate the outcome distribution at one exact time. It summarizes observations across the times and protocols represented in the bin. If the model-predicted distribution changes appreciably across that interval, or if protocols have different time patterns, the bin summary can combine unlike distributions. This is why the design and the definition of the target need to be considered together.
Match the Observed and Simulated Design
For a standard design-conditional VPC, hold the observed sampling design fixed when generating each replicate: use the corresponding individuals’ observation times and protocol assignments, and apply the same bin boundaries and summary calculation to simulated and observed data. This makes the comparison about outcomes under the design that produced the observations, rather than about outcomes under a different sampling schedule.
The observed times need not be evenly spaced. If a bin contains many observations, its empirical quantiles will generally be more stable than quantiles in a bin containing only a few observations. The number of observations is therefore important for interpreting the resolution and uncertainty of the display; it is not a reason to silently change the target or to compare against simulations with a different composition.
This approach assumes that conditioning on the recorded design is appropriate for the predictive question. If observation times are chosen in response to evolving concentrations, symptoms, or other information related to the outcome, the timing process may be informative. Merely reusing the recorded times in simulation does not necessarily account for that dependence. The analysis may then need a model for the observation process or a clearly restricted target that conditions on the relevant information. The interpretation should state which target is being checked.
Worked Example: Uneven Counts Across Time Bins
Suppose a hypothetical study has four observations in an early bin and eight in a later bin. The sorted observed concentrations in the early bin are 2, 3, 5, and 6 mg/L; in the later bin they are 1, 2, 2, 3, 3, 4, 5, and 6 mg/L. Use the nearest-rank empirical quantile convention: the quantile at proportion \(p\) is the observation with rank \(\lceil np\rceil\) in the sorted sample.
For the early-bin median, \(n=4\), so the selected rank is \(\lceil 4(0.5)\rceil=2\). The observed median is therefore 3 mg/L. For the later-bin median, \(n=8\), the rank is \(\lceil 8(0.5)\rceil=4\), giving an observed median of 3 mg/L.
These medians happen to agree, but they do not have equal precision: one is calculated from four observations and the other from eight. In the VPC, each simulated replicate must use the same four early-bin and eight later-bin observation slots as the data, then calculate medians with the same convention. A wider simulated reference envelope in the early bin would be consistent with its lower information, not automatically a model discrepancy. The counts alone do not prove that the model is adequate; the observed summaries must still be assessed against the properly generated simulations.
What a Pooled Quantile Means
When protocols contribute unequally to a bin, the pooled empirical distribution gives more weight to protocols with more observations. Let \(g\) index protocols, let \(n_g\) be the number of observations from protocol \(g\) in the bin, and let \(F_g(y)\) be that protocol’s empirical cumulative distribution function. The pooled empirical distribution is the observation-weighted mixture:
The pooled quantile is calculated from this combined distribution. In general, it is not the weighted average of the protocol-specific quantiles. Quantiles are nonlinear summaries: changing the mixture weights can move the pooled quantile in ways that cannot be recovered by averaging the separate quantile values.
Worked Example: Equal Protocol Weights Do Not Mean Averaging Medians
Consider two hypothetical protocols, A and B, with four observations each in a bin. All four values in A are 0 mg/L, and all four values in B are 10 mg/L. Use the nearest-rank convention for the median.
The protocol-specific medians are 0 mg/L for A and 10 mg/L for B. Their arithmetic average is \((0+10)/2=5\) mg/L. But the equally weighted pooled empirical distribution is \(F(y)=0.5F_A(y)+0.5F_B(y)\). At 0 mg/L, \(F(0)=0.5(1)+0.5(0)=0.5\). Thus the pooled median, defined as the smallest value where the cumulative distribution is at least 0.5, is 0 mg/L—not 5 mg/L.
Therefore, if the target is an equally weighted pooled quantile, combine the protocol-specific distributions with the stated weights and then calculate the quantile. Alternatively, report protocol-specific quantiles separately. Averaging protocol medians defines a different summary and must be labeled as such; it is not the equally weighted pooled median.
The same principle applies when the target is the observation-weighted pooled distribution. First combine the protocol-specific distributions using their observation shares, then calculate the quantile. Averages of protocol-specific quantiles should not be substituted for that calculation.
Choosing the Target When Protocols Differ
There is no universally correct weighting independent of the scientific question. An observation-weighted pooled target describes the distribution among the observations actually represented in the bin. It is often the natural target for a design-conditional VPC, provided simulations reproduce the observed protocol mix. An equally weighted protocol target instead gives each protocol the same contribution, regardless of its sample size. That can answer a useful question, but it is a different target and should be specified in advance.
For an equally weighted pooled target, combine the protocol-specific distributions with equal weights and calculate the desired quantile from the resulting mixture. Another option is to report protocol-specific summaries separately. If the goal is to standardize the observed and simulated data to a specified number of observations per protocol, apply the same standardization and summary procedure to both. Do not combine scalar protocol quantiles and describe their average as the pooled quantile.
| Target | How to calculate the bin summary | Interpretation |
|---|---|---|
| Observation-weighted pooled | Combine protocol distributions using each protocol’s share of observations, then calculate the quantile. | Summary for the observation mix represented in the bin. |
| Equally weighted pooled | Combine protocol distributions with equal protocol weights, then calculate the quantile. | Summary for a target that gives each protocol equal distributional weight. |
| Protocol-specific | Calculate and display a separate summary for each protocol. | Shows whether predictive performance differs by protocol. |
Worked Example: A Protocol-Mix Mismatch Can Create a False Signal
In a hypothetical time bin, protocol A contributes eight observed concentrations: 1, 2, 3, 4, 5, 6, 7, and 8 mg/L. Protocol B contributes two: 20 and 30 mg/L. Assume the fitted model reproduces these protocol-specific distributions, so the simulated values have the same distributions when the same design is used.
For the observation-weighted pooled target, sort all ten observations. The fifth value is 5 mg/L, and the nearest-rank median uses rank \(\lceil 10(0.5)\rceil=5\); the pooled median is 5 mg/L. A design-matched simulation also has eight A observations and two B observations, so it targets this same mixture.
Now suppose the observed data are compared with simulations that incorrectly give each protocol four observations. For an equally weighted pooled distribution, each protocol contributes half of its own distribution. The cumulative probability at 8 mg/L is \(0.5(1)+0.5(0)=0.5\), so the equally weighted pooled median is 8 mg/L. A comparison of the observed observation-weighted median, 5 mg/L, with this simulation target could look like a discrepancy even though the protocol-specific model predictions are correct.
The correction depends on the intended question. For the observation-weighted target, retain the 8:2 mix in both observed and simulated summaries. For an equally weighted target, construct the same equal-weight mixture for the observed and simulated protocol distributions and calculate each pooled quantile from that mixture. Averaging protocol medians would not calculate either pooled median in general.
Diagnostics and Failure Modes
When a VPC appears to diverge in a region with irregular sampling, first check the construction of the comparison before attributing the pattern to the model. The following checks are particularly useful:
- Display bin counts and protocol composition. A time bin with very few observations can yield unstable empirical quantiles. A change in protocol shares across bins can also change the pooled target.
- Verify the simulated design. Confirm that replicate data use the observed observation times, protocol assignments, and bin membership relevant to the chosen target.
- Check the bin’s time span. If predictions vary substantially over a wide interval, the bin pools outcomes from distinct times. Narrowing or otherwise revising the bin can clarify the time-specific pattern, but may reduce the number of observations and precision.
- Inspect protocol-specific summaries. A pooled discrepancy may be driven by one protocol, or may disappear when the compared protocol composition is made consistent. Separate displays can reveal this distinction.
- Review how observations entered the data. Missed samples, deviations from scheduled times, or outcome-dependent sampling can change the observed design. The implications depend on whether the timing process is informative for the predictive target.
A common mistake is to treat a dense region of the time axis as stronger evidence simply because the plot contains more points, or to interpret a sparse-bin quantile as equally precise. The correction is to examine the simulated reference behavior and the bin counts together: more observations generally improve empirical resolution, while fewer observations make the observed summary more variable. Neither density nor sparsity alone establishes model adequacy.
A second mistake is to “balance” protocols by averaging their medians and compare that number with an ordinary pooled median. Those are different estimands. Define the intended weighting first; then use the same distributional weighting and quantile calculation for the observed and simulated data. If the goal is an average of protocol-specific medians, report it explicitly as that average rather than calling it a pooled median.
Practical Interpretation
A VPC with irregular sampling is most interpretable when its target is explicit. For each bin, identify the observation times included, the contributing protocol mix, and whether the summary is observation-weighted, equally weighted, or protocol-specific. Then ensure the simulations reproduce that target. This prevents uneven observation density from being mistaken for a change in model performance and makes genuine predictive discrepancies easier to identify.
There are limits to what binning can resolve. Wider bins can improve the number of observations contributing to a quantile but may conceal changes across time; narrower bins can better localize a pattern but produce unstable summaries. If the timing mechanism itself carries information about outcomes, a design-conditional VPC may not address the full predictive question. In that case, clarify the target and consider whether the observation process must be represented rather than assuming that matching recorded times is sufficient.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A bin contains 12 observations from protocol A and 3 from protocol B. What protocol weights define the observation-weighted pooled distribution?
- Two protocols have medians of 4 and 12 mg/L. Is their average necessarily the equally weighted pooled median? Why or why not?
- A sparse-bin observed median lies outside the central region of a VPC. Name two checks to make before concluding that the model has a time-localized discrepancy.
- If the target is an equally weighted pooled quantile, how should the protocol distributions be combined?
- Why may reusing observed times in simulations be insufficient when observation timing depends on evolving outcomes?
Solutions. (1) The weights are \(12/15=0.8\) for A and \(3/15=0.2\) for B. (2) No. Quantiles are nonlinear; combine the two distributions with equal weights and calculate the quantile, or report the average medians as a distinct summary. (3) Check the bin count and simulated reference behavior, then verify that simulations use the same times, binning, and protocol mix; also inspect whether a wide bin combines changing predictions. (4) Form the equally weighted mixture \(F(y)=0.5F_A(y)+0.5F_B(y)\), then find its quantile. (5) If outcomes affect when samples are taken, conditioning on recorded times may omit an informative observation process; the target may require modeling that process or explicitly conditioning on additional information.