Why Examine the Extremes?
Prior predictive checks ask what trial outcomes a model and its prior could generate before the trial data are observed. Predictive quantiles, covered in the earlier material on prior predictive quantiles, summarize much of the distribution’s central range. But a model can have plausible central quantiles and still assign substantial weight to outcomes at its far edges. For example, a small fraction of simulated trials might have almost every participant experience an event, or might show a treatment contrast far beyond a clinically credible range.
A tail diagnosis is not just a search for the most unusual simulated value. It asks three related questions: what extreme outcomes are generated, how often do they occur, and which parameter values tend to generate them? The last question matters because a surprising outcome can arise from an implausible parameter draw, ordinary sampling variation, or a combination of both.
The underlying calculation is a mixture: the predictive probability of an extreme outcome averages its conditional probability over the prior distribution of parameters. For any region \(A\) of parameter space, this can be split into the contributions from parameter values inside and outside that region.
This decomposition gives a practical diagnostic. If extreme outcomes occur mainly when \(\theta\) falls in a clinically implausible region, the prior may be allowing implausible parameter values. If they occur across ordinary parameter values, the sampling model may naturally produce a large spread, or the outcome threshold may need reconsideration. Either way, the simulated outcome should be traced back to the parameter draw that generated it.
How to Inspect a Predictive Tail
For each simulation replicate, retain both the parameter draw and the resulting trial outcome. Then define an extreme-outcome region before inspecting the most extreme draws. For a one-sided concern, this might be \(Y\ge c\); for a two-sided concern, it might be \(|Y|\ge c\). Estimate the frequency by dividing the number of replicates in the region by the total number simulated. This is a descriptive estimate of the model’s predictive probability, not a probability that the prior or model is true.
Define the trial-level quantity to inspect, its units, and a concerning range using clinical reasoning.
For each replicate, draw from the prior and then from the sampling model, keeping the paired parameter and outcome values.
Report their number and fraction, then examine whether they cluster at particular parameter values or combinations of values.
Decide whether the frequency and source of the extremes are compatible with the intended clinical model, and document the reasoning.
The assumptions are those of the specified prior and sampling model. Conditional outcome distributions must be appropriate for the trial design, follow-up, endpoint, and dependence structure. For instance, treating event indicators as independent conditional on a risk parameter is not justified if the model includes unrepresented clustering. A diagnostic can reveal a problematic implication of a model, but simulation cannot repair a misspecified model.
Worked Example: Near-Universal Events Under a Beta Prior
Consider a hypothetical single-arm trial with \(n=100\) participants. Suppose the event risk satisfies \(p\sim\operatorname{Beta}(1,1)\), and, conditional on \(p\), the event count \(K\) satisfies \(K\mid p\sim\operatorname{Binomial}(100,p)\). Assume participants’ outcomes are conditionally independent given \(p\). As a deliberately concerning diagnostic region, examine \(K\ge95\).
Integrating over the uniform prior gives a beta-binomial predictive distribution. For each possible count \(k\),
Thus,
Under this model, about 5.94% of prior predictive trials have at least 95 events. That frequency is a potentially important implication if such an event burden is clinically implausible. It is not enough, however, to see that the event count is extreme; the parameter values that generate it should also be checked.
For the most extreme outcome, \(K=100\), the conditional distribution of \(p\) is proportional to the likelihood times the prior:
Moreover, the conditional probability that \(p>0.95\) is \(1-0.95^{101}\), approximately 0.9944. So an all-event trial is overwhelmingly associated with a very high prior risk draw. The prior’s uniform distribution on the risk scale allows such values; this diagnostic identifies that implication. Whether those values are clinically credible must be judged using the endpoint and intended population.
Worked Example: A Wide Prior Predictive Treatment Estimate
Suppose a hypothetical trial estimates a treatment contrast \(Y\), measured in points, with model \(Y\mid\theta\sim N(\theta,3^2)\). Here \(\theta\) is the true contrast, and the sampling standard deviation of the estimate is 3 points. Let \(\theta\sim N(0,8^2)\), independently specified before observing the trial. Assume this normal sampling model and prior are appropriate for the contrast. Suppose contrasts outside \(-15\) to \(15\) points are already a concern, and values beyond 20 points are especially extreme.
The predictive distribution follows by adding the variance of the prior contrast and the conditional sampling variance. Therefore \(Y\sim N(0,8^2+3^2)=N(0,73)\), with predictive standard deviation \(\sqrt{73}=8.544\) points. The probability of an estimate beyond 20 points in either direction is
So about 1.92% of estimates fall beyond 20 points under this prior predictive model. The broader concern threshold of 15 points is exceeded with probability \(2\Phi(-15/8.544)\approx 0.0793\), or 7.93%. These probabilities describe the model’s implications for trial estimates, not the chance that a treatment effect is real.
To see what generates an estimate of \(Y=25\) points, the conditional distribution of \(\theta\) given \(Y\) is normal. Completing the square in the product of the normal prior density and the normal sampling density gives
The conditional standard deviation is \(\sqrt{7.890}=2.809\) points. Hence \(P(\theta>15\mid Y=25)=\Phi((21.92-15)/2.809)\approx0.9931\). An estimate of 25 points is therefore associated with a very large underlying contrast, rather than being explained mainly by a typical \(\theta\) plus ordinary sampling noise. The prior places probability \(P(\theta>15)=\Phi(-15/8)=0.0304\) on that one-sided parameter region. This comparison helps locate the source of the extreme, while recognizing that the outcome and parameter are not identical.
Worked Example: An Extreme Survival Event Count
Consider a hypothetical study with 20 participants followed for 12 months, no competing risks, and independent event times conditional on a common hazard \(\lambda\). Use an exponential event-time model and the prior \(\log(\lambda)\sim N(\log(0.05),1.5^2)\), with hazard measured per month. Let \(K\) be the number of participants with an event by 12 months. For illustration, suppose \(K\ge19\) would be clinically alarming.
Conditional on \(\lambda\), each participant’s event probability by month 12 is \(q(\lambda)=1-\exp(-12\lambda)\), and \(K\mid\lambda\sim\operatorname{Binomial}(20,q(\lambda))\). The prior median hazard is 0.05 per month, but its log-scale standard deviation is large. The prior probability of \(\lambda\ge0.5\) is
At \(\lambda=0.5\), the event probability is \(q=1-\exp(-6)=0.9975\). Thus the conditional probability of at least 19 events is
For larger hazards this probability is at least as large, because \(q(\lambda)\) increases with \(\lambda\). Consequently, the parameter region \(\lambda\ge0.5\) alone contributes at least \(0.0624\times0.9989=0.0623\) to the prior predictive probability of \(K\ge19\). This is a lower bound: smaller hazards may also contribute. The calculation shows how a right tail in the hazard prior can generate near-universal events by the specified follow-up time. Whether that hazard region is plausible depends on the disease, endpoint, and follow-up context.
What to Look For—and What Not to Conclude
A useful review does not stop at the largest simulated value. Examine a set of extreme replicates and compare their parameter draws with the rest of the simulations. Look for a tail driven by a small number of very large parameter values, by a particular combination of parameters, or by outcomes that remain extreme under otherwise ordinary parameter values. In hierarchical models, retain the relevant group-level and shared parameters as well as the trial outcome; otherwise the feature responsible for the extreme may be hidden.
The following distinctions help prevent an unhelpful diagnosis:
| Observation | Possible implication | Diagnostic question |
|---|---|---|
| Extreme outcomes occur alongside extreme parameter draws. | The prior may allow parameter values that conflict with clinical knowledge. | Are those parameter values credible for this population and endpoint? |
| Extreme outcomes occur with ordinary parameter draws. | Sampling variation, the model structure, or the outcome definition may be responsible. | Does the conditional sampling model match the design and dependence structure? |
| Only a few extreme replicates appear. | The tail may be rare, or the simulation may be too small to characterize it reliably. | Is the observed frequency stable enough to support the intended conclusion? |
A common mistake is to interpret a central predictive interval as a guarantee that outcomes outside it cannot occur, or to dismiss extremes because they form a small percentage of simulations. The correction is to distinguish central coverage from the existence and clinical relevance of a tail. A 95% interval leaves 5% of the predictive distribution outside its bounds by construction; that remainder may still include outcomes with serious clinical implications. Conversely, a simulated maximum is not itself evidence of a defective prior: maxima tend to grow as more simulations are run, so judge them against a prespecified clinical region and their frequency.
Another pitfall is to blame every extreme outcome on a heavy-tailed prior. A tail can arise from the likelihood even when parameter values are moderate, and a prior that looks concentrated on its own scale may imply a broad range of outcomes after the trial design and follow-up are applied. Inspecting paired parameter and outcome draws helps separate these explanations. This approach complements, rather than replaces, predictive quantiles: quantiles describe a chosen portion of the distribution, while tail inspection asks what happens in scientifically concerning regions beyond the central summary.
The diagnostic has limits. Its conclusion depends on the clinical threshold, the chosen outcome summary, and the model assumptions. Simulation can miss very rare events, and a finite set of replicates cannot establish that an event is impossible. It also cannot determine which prior is scientifically correct. The purpose is to make the model’s implications visible and to decide whether they warrant closer clinical review.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a trial model, what information must be retained during simulation to diagnose which parameter values generate extreme outcomes?
- For \(K\mid p\sim\operatorname{Binomial}(n,p)\), explain why a large event count can be due to both the risk parameter and sampling variation.
- A predictive estimate is extreme even though its associated parameter draw is near the center of the prior. Name two explanations to investigate.
- Why is a simulated maximum not, by itself, a sound criterion for rejecting a prior?
Solutions. (1) Retain each simulated parameter draw paired with its resulting trial outcome. (2) The value of \(p\) determines the conditional event probability, but even for fixed \(p\), binomial sampling can produce counts both above and below its expectation \(np\). (3) Check the conditional sampling variation and whether the sampling model, design assumptions, or outcome definition are appropriate. (4) The maximum can increase simply because more replicates are simulated; instead define a clinically concerning region and assess its frequency and generating parameter values.