Common Misreadings Of Prior Predictive Checks

What a Prior Predictive Check Can—and Cannot—Tell You

A prior predictive check asks what trial data a proposed model could generate before any trial outcomes are observed. It is a way to examine the implications of the prior distribution together with the sampling model and planned design. The check can reveal that the model generates implausible event counts, treatment differences, or other trial summaries. But a simulated outcome is not evidence about what actually happened, and an individual simulation is not a verdict on the prior.

Many misreadings arise from losing track of what is being generated. A prior draw is a possible value of an unknown parameter. A prior predictive replicate is a possible data set generated using that draw and the trial model. The replicate is the object to compare with clinically meaningful outcomes or benchmarks. Both layers matter, but they answer different questions.

Definition: Let $\theta$ denote model parameters, $d$ the planned trial design, and $Y$ the trial data. The prior predictive distribution is $$ p(y\mid d)=\int p(y\mid\theta,d)\,p(\theta)\,d\theta. $$ It describes the distribution of possible data under the assumed prior, sampling model, and design, before conditioning on observed trial data. A prior predictive replicate is generated by first drawing $\theta\sim p(\theta)$, then drawing $Y^{\mathrm{rep}}\sim p(y\mid\theta,d)$.

The design belongs in this definition. Sample size, allocation, follow-up, outcome definition, and relevant dependence assumptions all affect the generated data. A simulation that uses the wrong design can look like a check while answering a different question. A check also depends on the likelihood being used: plausible parameter values cannot guarantee plausible data if the sampling model omits important features.

Misreading 1: Treating Simulations as Observed Evidence

A prior predictive replicate is generated by the model; it is not a new observation from the clinical trial. If a simulated trial has few events, that does not show that few events occurred. Nor does a replicate with a large treatment difference show that the treatment is effective. The replicate illustrates one possible outcome under the specified assumptions.

The relevant evidence about a real trial comes from its observed data. Before those data are available, prior predictive checks instead support a different task: deciding whether the proposed assumptions generate outcomes that are reasonable enough to use for the planned analysis. For instance, a prior that regularly generates control-arm event rates far outside the clinically credible range may need reconsideration. That conclusion concerns the model’s implications, not the truth of a particular simulated rate.

Worked Example: A Simulated Event Count Is Not Trial Evidence

Suppose a hypothetical trial plans to enroll $n=100$ participants in a control arm. The model assigns the control event probability $p_C\sim\operatorname{Beta}(2,8)$, and, conditional on $p_C$, the count of participants with the event is $K\mid p_C\sim\operatorname{Binomial}(100,p_C)$. A prior predictive run draws $p_C=0.08$, then generates $K=8$.

The simulated rate is $8/100=0.08$, or 8 events per 100 participants. This is a possible dataset produced by the model. It is not an observed rate from a real trial, and it does not provide evidence that the actual control event rate is 8%.

The prior predictive distribution itself can be summarized. For $p_C\sim\operatorname{Beta}(a,b)$, the mean is $a/(a+b)$ and the variance is $ab/\{(a+b)^2(a+b+1)\}$. Thus, here, $$ E(p_C)=\frac{2}{10}=0.20,\qquad \operatorname{Var}(p_C)=\frac{2(8)}{10^2(11)}\approx0.01455. $$

The count’s predictive mean is $E(K)=100E(p_C)=20$. To obtain its predictive variance, use the law of total variance: $$ \operatorname{Var}(K)=E\{\operatorname{Var}(K\mid p_C)\}+\operatorname{Var}\{E(K\mid p_C)\}. $$ Since $\operatorname{Var}(K\mid p_C)=100p_C(1-p_C)$ and $E(K\mid p_C)=100p_C$, first calculate $$ E\{p_C(1-p_C)\}=E(p_C)-E(p_C^2) =0.20-\frac{2(3)}{10(11)} \approx0.14545. $$ Therefore, $$ \operatorname{Var}(K)=100(0.14545)+10000(0.014545)\approx160, \qquad SD(K)\approx\sqrt{160}=12.65. $$

The predictive mean is 20 events and the predictive standard deviation is about 12.65 events. These summaries describe variation across generated trials, not uncertainty about whether this particular simulated count occurred in a real study. The calculation also shows why it is important to check arithmetic at both levels: the first variance term reflects binomial sampling, while the second reflects variation in the prior event probability.

Misreading 2: Checking Parameters Instead of Data

Inspecting prior parameter draws can be useful, but it is not by itself a prior predictive check. A parameter has meaning through the model and design that turn it into observable outcomes. A prior value that looks numerically moderate may generate extreme data for a large trial; conversely, a wide parameter distribution can still imply a reasonable range of outcomes for a small trial. The outcome scale and sample size matter.

For example, reviewing draws of $p_C$ may reveal whether the prior puts mass on control event rates that clinicians consider implausible. But the prior predictive question goes further: what event counts or rates arise after those probabilities are passed through the planned binomial sampling model? The distinction is especially important when the check concerns treatment contrasts, rare events, correlated endpoints, or subgroup summaries. The generated data must preserve the trial structure relevant to the question.

Worked Example: Keeping the Sign and Scale of a Treatment Contrast Straight

Consider a hypothetical two-arm trial with 100 participants per arm. In one prior predictive replicate, suppose the parameter draws are $p_C=0.08$ and $p_T=0.18$. The generated counts are 8 control events and 10 treatment events. Define every difference as treatment minus control.

The generated proportions are $8/100=0.08$ in control and $10/100=0.10$ in treatment. Thus, the generated treatment-minus-control difference is $$ 0.10-0.08=0.02. $$ Conditional on the parameter draws, the expected difference is $p_T-p_C=0.18-0.08=0.10$. The replicate’s deviation from that conditional expectation is therefore $$ 0.02-0.10=-0.08. $$

Under independent binomial sampling in the two arms, the conditional standard error of the difference in proportions is $$ SE=\sqrt{\frac{p_C(1-p_C)}{100}+\frac{p_T(1-p_T)}{100}} =\sqrt{\frac{0.08(0.92)}{100}+\frac{0.18(0.82)}{100}} \approx0.04703. $$ So the replicate’s difference is about $-0.08/0.04703\approx-1.70$ conditional standard errors below its conditional expectation. This describes the replicate relative to the fixed parameter draws; it does not redefine the contrast or imply that the prior expected treatment effect is negative. In particular, the difference remains $+0.02$, because the convention is treatment minus control.

This calculation is a conditional diagnostic for one generated replicate, not a complete assessment of the marginal prior predictive distribution. A full check considers the distribution of replicated contrasts over prior draws and sampling variation, and compares it with clinically relevant ranges. It also verifies that the contrast’s direction, units, and arm labels are consistent throughout.

Misreading 3: Judging Plausibility from One Replicate

A single replicate can be unusually high or low even when generated by a reasonable model. Conversely, a single ordinary-looking replicate does not establish that the model is adequate. The check concerns the distribution of possible outcomes, so examine many replicates and summarize features that address the clinical question: for example, predictive quantiles, the frequency of a clinically important event, or the range of treatment differences.

This does not mean that every extreme replicate can be ignored. Repeatedly generating outcomes outside a credible range, or assigning substantial prior predictive probability to a clinically impossible outcome, is a reason to investigate the assumptions. The key is to assess how often and under what conditions the feature occurs, rather than treating one generated dataset as decisive.

Worked Example: One Low Count in a Wide Predictive Distribution

Return to the hypothetical control arm with $n=100$ and $p_C\sim\operatorname{Beta}(2,8)$. The predictive count has mean 20 and standard deviation about 12.65. Suppose one replicate has $K=8$ events.

Relative to the predictive mean and standard deviation, this count is $$ \frac{8-20}{12.65}\approx-0.949. $$ It is about 0.95 predictive standard deviations below the mean. That calculation is a scale reference, not a tail probability: the beta-binomial predictive distribution is discrete and need not be symmetric or normal. It does, however, show why calling 8 events implausible merely because it is below 20 would be misleading.

A useful check would examine the whole predictive distribution, including quantiles or a prespecified clinically relevant threshold such as fewer than a chosen number of events. If many replicates fall in a region clinicians regard as incompatible with the trial setting, the prior or model deserves review. If only an occasional replicate is there, that frequency—not the isolated draw—determines how much attention to give it.

Assumptions, Diagnostics, and Alternatives

Interpret a prior predictive check only after stating what generated the data. At minimum, confirm that the prior is the one intended, the likelihood matches the endpoint, and the simulation represents the planned design. For a binary endpoint, this includes the number assigned to each arm and the assumed dependence structure. For a time-to-event or repeated-measures endpoint, the generated summaries must reflect follow-up and within-participant structure relevant to the check. A mismatch can create alarming or reassuring outputs for the wrong reason.

Common warning signs include swapped arm labels, a reversed treatment-effect convention, summaries on an unclear scale, and comparison of a parameter to a benchmark intended for observed outcomes. Check that the replicated quantity is actually the quantity named in the clinical question. If a benchmark is a rate per 100 participants, for example, do not compare it directly with a count from a trial of a different size without converting or otherwise accounting for the design.

Prior predictive checks are complementary to, not interchangeable with, other ways of assessing assumptions. Reviewing parameter priors can expose implausible parameter values; clinical elicitation can help specify credible ranges; and sensitivity analysis can show whether conclusions depend on particular prior choices. None replaces checking the implications for observable trial data. Conversely, predictive agreement alone does not prove that a prior is clinically justified or that the likelihood captures the real data-generating process.

Key takeaway: A prior predictive check evaluates the distribution of data the model can generate under the planned design. Simulations are hypothetical outcomes, parameter draws are not data, and one replicate is not a verdict. Interpret the distribution, the clinical benchmark, and the assumptions that generated it together.

Check Your Understanding

Answer each question by distinguishing parameter values, generated data, and the distribution across replicates.

A prior predictive simulation generates 14 events in a hypothetical control arm. Does this establish that 14 events occurred in the actual trial? Explain.
Why is examining draws of a response probability alone not a complete prior predictive check?
In a trial, a generated control proportion is 0.12 and a generated treatment proportion is 0.17. Under the treatment-minus-control convention, what is the generated difference?
Why can one extreme generated replicate be insufficient reason to reject a model?
Name two features that should be verified before interpreting a prior predictive summary.

Solutions. (1) No. The count is a hypothetical output of the prior and sampling model, not observed trial evidence. (2) Parameter draws have not yet been translated into data through the likelihood and design; sample size and sampling variation affect observable outcomes. (3) The difference is $0.17-0.12=0.05$. (4) A replicate is one possible draw; plausibility depends on how often relevant outcomes arise across the predictive distribution and on the clinical benchmark. (5) Examples include the prior and likelihood used, the planned sample size and allocation, arm labels and contrast direction, and whether the generated summary matches the benchmark’s units.