Two Sources of Variation in a Predictive Check
A predictive check compares features of the observed data with features of data generated under a fitted model. An apparent discrepancy can be difficult to interpret: is the observed feature unusual under the model, or has the reference comparison itself been estimated unreliably from a finite number of simulations? These are different questions. The first concerns predictive variability; the second concerns Monte Carlo uncertainty.
For a chosen predictive-check summary, each simulated data set gives one value. The distribution of those values describes how much that summary varies across data sets generated under the model and study design. A predictive envelope, such as the 5th to 95th percentiles of the simulated summaries, describes that spread. It does not, by its width alone, show how precisely the envelope limits have been estimated. Precision of the estimated limits must be assessed separately.
The interpretation depends on a clear simulation setup. In this tutorial, simulations are independent draws under a specified fitted model, using the same design and the same predictive-check calculation as for the observation. The model parameters are held fixed at their fitted values. A conditional check must use the same conditioning information for the observed and simulated summaries; an unconditional check must likewise simulate and summarize consistently. The resulting comparison is conditional on that setup. It does not automatically include uncertainty in the fitted parameters.
The summary and its scale should be chosen before judging the results. For example, \(S\) could be a median concentration in a time bin, a proportion below a limit, or an NPDE summary. A value outside an envelope indicates that the observed summary is unusual relative to the simulated predictive distribution at that location and scale. It is not, by itself, a complete diagnosis of why the discrepancy occurred.
Predictive Spread Is Not Precision of the Estimated Limits
Suppose the 5th and 95th percentiles of \(S^{(m)}\) are used as envelope limits. If the simulated summaries are widely dispersed, the envelope will be wide. That width describes predictive variation in the summary across generated data sets. If the summaries are tightly dispersed, the envelope will be narrow. Neither result, on its own, indicates whether the estimated limits are precise: even a narrow envelope can have poorly estimated limits when \(M\) is small, while a broad envelope can have accurately estimated limits when \(M\) is large.
For independent simulations, one useful diagnostic is to divide the simulations into separate batches and calculate the same quantile in each batch. Substantial variation among batch quantiles suggests that the finite-simulation estimate may be unstable. Similar batch quantiles are reassuring, though they are not a formal guarantee of precision. Alternatively, a quantile-specific standard error can be estimated using a suitable method. Such an estimate depends on the density of the simulated-summary distribution near the quantile: quantiles in sparse or irregular regions can be difficult to estimate precisely.
For a tail probability, Monte Carlo precision has a direct calculation. If \(I_m\) equals 1 when \(S^{(m)}\) falls in the specified tail region and 0 otherwise, then \(\hat p=M^{-1}\sum_{m=1}^M I_m\) estimates the model probability \(p\) of that event. Under independent simulations, its approximate Monte Carlo standard error is \(\sqrt{\hat p(1-\hat p)/M}\). This quantifies simulation error in the estimated tail probability, not the predictive spread of the underlying summary values.
Why Repeated Simulation Helps
The distinction can be made precise for a predictive quantile. For a continuous predictive distribution with \(p\)-quantile \(q_p\), each simulated summary has probability \(p\) of falling below \(q_p\). Among \(M\) independent simulated summaries, the number below \(q_p\) therefore has a binomial distribution with mean \(Mp\) and variance \(Mp(1-p)\). The estimated quantile is consequently subject to rank variation even though the underlying model and predictive distribution have not changed.
This derivation explains why repeated batches are informative: different batches can select different simulated order statistics as the estimated quantile. The rank calculation gives a way to understand the scale of this fluctuation, but it does not directly give uncertainty in the summary's measurement units. To assess uncertainty in the quantile value, inspect batch quantiles or use an appropriate quantile-specific standard error.
A practical interpretation proceeds in two stages. First, compare the observed summary with the estimated predictive distribution. Second, check whether the relevant predictive limits or tail estimates are stable enough to support that comparison. A discrepancy that appears only under one batch's limits, but disappears under equally valid batches, is weak evidence of persistent model misfit. A discrepancy that remains in the same direction and of similar size under repeated simulation is less likely to be an artifact of the particular finite set of replicates.
Worked Example: Observed Median Above a Predictive Envelope
Hypothetical example. In one concentration-time bin, the observed median is \(12.3\) mg/L. A fitted model with a fixed design generates \(M=2{,}000\) independent data sets, and the median is calculated in the same way for each. The estimated 5th, 50th, and 95th percentiles of the simulated medians are \(8.4\), \(10.1\), and \(12.0\) mg/L.
The estimated 5th-to-95th-percentile predictive envelope is \(8.4\) to \(12.0\) mg/L. The observed median, \(12.3\) mg/L, is \(12.3-12.0=0.3\) mg/L above the estimated upper limit. Thus, under this fitted model and simulation setup, the observed bin median is above the estimated central 90% predictive range.
This comparison does not show that the upper limit is estimated exactly. Suppose five independent batches of 400 simulations give 95th-percentile estimates of \(11.8\), \(12.1\), \(12.2\), \(12.0\), and \(12.1\) mg/L. The observed median remains above every batch estimate, by between \(0.1\) and \(0.5\) mg/L. That consistency makes it less plausible that this particular finding is caused solely by which simulations happened to be drawn. It still does not identify a cause: bin construction, the fitted model, and other features of the check may need review.
The envelope width, \(12.0-8.4=3.6\) mg/L, describes the spread between the estimated predictive quantiles. The range of batch-specific upper limits describes a different feature: how much the estimated upper limit varied across these batches. One cannot infer the second quantity from the first.
Assessing Monte Carlo Precision in Practice
When \(M\) is limited or a result is near an envelope boundary, assess the precision of the reference estimate directly. Independent batches are easy to interpret because each gives a separate estimate from a disjoint set of simulations. Use the same model, design, summary definition, and quantile probability in every batch. Compare the batch estimates in the original units, and, where relevant, compare the observed summary with each batch's envelope.
Batch agreement is a diagnostic, not a substitute for careful quantile estimation. A formal quantile standard error can be useful when its assumptions and implementation are appropriate. In particular, an asymptotic quantile standard error relies on enough simulations and a reasonably well-behaved density near the target quantile. Discrete summaries, ties, very sparse tails, and multimodal distributions can make that approximation unreliable; batch comparisons or additional simulations may be more informative.
For a tail probability, the standard error formula gives a quick scale check. If the estimated tail probability is near 0.05 from \(M=1{,}000\) independent replicates, the approximate Monte Carlo standard error is about 0.0069. A change of a few thousandths across simulation runs would not be surprising at that simulation count. This calculation concerns uncertainty in the estimated probability; it does not say how far the simulated summary values themselves typically vary.
Worked Example: Checking a Tail Estimate With Batches
Hypothetical example. In a chosen bin, define a lower-tail event as a simulated summary at or below the observed summary. Across \(M=1{,}000\) independent simulations, 47 summaries fall in that event. The estimated predictive tail probability is
The approximate Monte Carlo standard error is
Thus the estimated tail probability is 0.047, with Monte Carlo standard error about 0.0067. The standard error describes the finite-simulation uncertainty in estimating the tail probability under the assumed independent-replicate setup. It is not the standard deviation of the simulated summaries, and it does not mean that the model probability itself varies across batches.
As a separate check, divide the simulations into four batches of 250. Suppose the numbers in the lower-tail event are 10, 13, 11, and 13. The batch estimates are \(10/250=0.040\), \(13/250=0.052\), \(11/250=0.044\), and \(13/250=0.052\). These are broadly consistent with the pooled estimate \(0.047\), given the smaller batch sizes. The batch comparison supports the view that the pooled tail estimate is not dominated by one unusual batch; it does not prove that the model fits well.
Persistent Patterns, Diagnostics, and Alternatives
Predictive checks often involve multiple bins or summaries. A pattern is more persuasive as a model diagnostic when its direction and location persist across independent simulation batches, rather than appearing only because one set of replicates produced a particular envelope. For example, repeated observed medians above their upper reference limits across adjacent time bins may indicate a systematic predictive discrepancy. But the bins can be dependent, and examining many bins creates opportunities for chance excursions. Repetition across batches addresses Monte Carlo stability; it does not by itself correct for multiple comparisons or establish a formal test.
Specify the observed summary, simulation design, conditioning choice, and reference quantiles before comparing results.
Use the distribution of replicate summaries to form the intended predictive envelope or tail probability.
Compare independent batch estimates or use a suitable quantile-specific precision method. Increase the number of simulations when the estimate is unstable or the comparison is borderline.
Describe which observed feature is unusual under the specified predictive distribution. Investigate the model and check construction without treating simulation stability as proof of model adequacy.
The main practical alternatives answer related but distinct questions. More simulations reduce Monte Carlo uncertainty in estimated reference features, but do not narrow the model's predictive distribution. Independent batches reveal run-to-run stability and are especially useful as a diagnostic. A quantile-specific standard error targets uncertainty in a quantile estimate, but can be unreliable for discrete or irregular distributions. A predictive envelope communicates the distributional spread of the simulated summary, but is not a precision interval for its own limits.
| Tool or quantity | What it describes | What it does not establish |
|---|---|---|
| Spread of simulated summaries | Predictive variability under the specified model and design | Precision of the estimated envelope limits |
| Independent batch quantiles | Stability of estimated limits across finite simulation runs | That the model is adequate if batches agree |
| Monte Carlo standard error for a tail proportion | Finite-simulation uncertainty in the estimated probability | Spread of summary values or parameter uncertainty |
| More simulations | Usually improves precision of estimated predictive features | Removal of model misspecification or parameter uncertainty |
Several warning signs call for caution: batch quantiles vary substantially; the observed value lies very close to an estimated limit; few simulations populate the tail; the summary is discrete with many ties; or the apparent pattern changes when the binning or summary definition changes. Simulation independence also matters. Reusing dependent replicates as if they were independent can make a nominal Monte Carlo standard error too small. Finally, all these checks are conditional on the fitted model and chosen simulation setup. A stable predictive discrepancy can flag incompatibility, but a stable comparison does not demonstrate that all relevant aspects of the model are correct.
Common Mistakes
- Calling envelope width Monte Carlo uncertainty. The width between predictive quantiles describes the spread of simulated summaries. Assess uncertainty in the estimated limits separately.
- Assuming a narrow band is precisely estimated. A narrow predictive envelope can still have unstable limits if there are too few simulations. Use batches or a suitable quantile precision method.
- Treating a stable result as proof of fit. Repeated simulations can show that a discrepancy is not merely a finite-simulation fluctuation. They cannot rule out misspecification elsewhere or account automatically for parameter uncertainty.
- Increasing simulations to solve model misfit. More simulations can stabilize the reference estimate, but do not make an observed discrepancy disappear systematically under the same model.
Check Your Understanding
Answer each question by identifying whether it concerns predictive variability, Monte Carlo uncertainty, or both.
- An envelope's 5th and 95th percentiles are far apart. What does that say, and what does it not say, about the estimated limits?
- For an estimated tail probability of 0.08 from 500 independent replicates, calculate the approximate Monte Carlo standard error.
- Why can comparing four independent batches help assess an estimated 95th percentile?
- An observed summary lies just above the pooled upper limit but below the upper limits in two of four batches. What is a cautious interpretation?
- Does doubling the number of simulations reduce the predictive spread of the summary? Explain.
Solutions. (1) The wide interval describes substantial predictive spread between those quantiles. It does not determine how precisely either quantile has been estimated; assess that with batches or a suitable quantile-specific method. (2) \(SE_{\mathrm{MC}}\approx\sqrt{0.08(0.92)/500}=\sqrt{0.0001472}\approx0.0121\). This is uncertainty in the estimated tail proportion. (3) Each batch gives a separate finite-simulation estimate; appreciable variation signals that the estimated limit may be unstable at the chosen simulation count. (4) The comparison is borderline and sensitive to finite-simulation variation, so it is not strong evidence of a persistent discrepancy without further simulation or precision assessment. (5) No. More simulations generally improve precision in estimating the predictive distribution; they do not change its underlying spread when the model and design are fixed.