Prior Predictive Checks Versus Posterior Predictive Checks

Two Predictive Checks, Two Questions

A predictive check asks whether outcomes generated by a model look plausible for the clinical question at hand. The word predictive refers to generating possible data under a probability model; it does not necessarily mean forecasting a future patient or trial. The key distinction in this tutorial is when the parameter values are drawn: prior predictive checks use the prior distribution, while posterior predictive checks use the posterior distribution after conditioning on observed trial data.

A prior predictive check asks whether the model and prior together imply outcomes that are clinically plausible before the trial results are used. A posterior predictive check asks whether outcomes replicated from the fitted model resemble features of the data that were observed. The checks are related, but they address different stages and should not be treated as interchangeable.

Definition: Let \(p\) denote a model parameter, \(\pi(p)\) its prior distribution, \(Y\) trial data, and \(y_{\mathrm{obs}}\) the observed data. The prior predictive distribution is the distribution of data generated by first drawing \(p\sim\pi(p)\), then drawing \(Y\) from the sampling model given \(p\). The posterior predictive distribution is the distribution of replicated data \(Y^{\mathrm{rep}}\) generated by first drawing \(p\) from the posterior distribution given \(y_{\mathrm{obs}}\), then drawing \(Y^{\mathrm{rep}}\) from the sampling model given \(p\).

In formulas, the prior predictive distribution averages the sampling model over the prior. The posterior predictive distribution averages it over the posterior. These integrals make clear that the model for generating trial outcomes is part of both checks.

p(y)=\int p(y\mid p)\,\pi(p)\,dp, \qquad p(y^{\mathrm{rep}}\mid y_{\mathrm{obs}}) =\int p(y^{\mathrm{rep}}\mid p)\,\pi(p\mid y_{\mathrm{obs}})\,dp.

Here \(p(y\mid p)\) is the sampling distribution, and \(\pi(p\mid y_{\mathrm{obs}})\) is the posterior distribution. In the equations, the italic \(p\) used as a parameter is distinct from the notation \(p(y)\) for a probability density or probability mass function; the intended meaning is determined by its argument.

A Binary-Response Trial Model

Consider a hypothetical single-arm clinical trial in which each of \(n\) participants either responds or does not respond by a prespecified assessment time. Let \(Y\) be the number of responses and \(p\) the response probability. Assume that, conditional on \(p\), participants’ responses are independent and have the same probability \(p\). Then \(Y\mid p\sim\operatorname{Binomial}(n,p)\). This model assumes a fixed assessment rule, complete outcome ascertainment, and no unmodelled differences in response probability across participants.

For illustration, assign \(p\sim\operatorname{Beta}(2,8)\). This prior has mean \(2/(2+8)=0.20\). A prior predictive check generates values of \(p\) from this beta distribution and then generates trial counts conditional on each value. The check therefore reflects both uncertainty about the response probability and binomial variation among participants.

Worked Example: Prior Predictive Center and Spread

Suppose the hypothetical trial plans to enroll \(n=20\) participants. Under the \(\operatorname{Beta}(2,8)\) prior, find the prior predictive mean and variance of the response count \(Y\).

Conditional on \(p\), \(E(Y\mid p)=np\) and \(\operatorname{Var}(Y\mid p)=np(1-p)\). Averaging the conditional mean over the prior gives

E(Y)=nE(p)=20\left(\frac{2}{10}\right)=4.

For a beta-binomial prior predictive distribution with beta parameters \(a\) and \(b\), the variance is \(nab(a+b+n)/[(a+b)^2(a+b+1)]\). Substituting \(a=2\), \(b=8\), and \(n=20\),

\operatorname{Var}(Y) =\frac{20(2)(8)(2+8+20)}{(2+8)^2(2+8+1)} =\frac{9600}{1100} =8.727.

The prior predictive standard deviation is \(\sqrt{8.727}=2.954\) responses. Thus, the model predicts an average of 4 responses in a trial of 20, with substantial spread. This calculation alone does not decide whether the prior is clinically acceptable; investigators must judge the implications against the intended population and endpoint.

What Each Check Conditions On

A prior predictive check is naturally used during model development and trial planning, before outcomes are available. Investigators can examine simulated counts, rates, or other prespecified summaries and ask whether the implied range and tails are credible. If a prior makes near-zero responses or near-universal responses common when those outcomes would be implausible, the model deserves review before it informs a trial decision.

A prior predictive check can also be compared with observed data after the trial. In that setting, the observed result is being compared with predictions that were generated without using that result. Such a comparison can reveal that the chosen prior-model combination did not anticipate the data. The observed data should not then be used to quietly revise the prior and still describe the original comparison as an untouched prior predictive check.

A posterior predictive check instead uses the observed data to update uncertainty about \(p\). For a hypothetical trial with 9 responses among 20 participants, the observed response proportion is \(\hat{p}=9/20=0.45\). Under the model above, Bayes’ rule gives a posterior proportional to the prior density times the binomial likelihood:

\pi(p\mid y_{\mathrm{obs}}) \propto p^{9}(1-p)^{11}p^{2-1}(1-p)^{8-1} =p^{10}(1-p)^{18},

so \(p\mid y_{\mathrm{obs}}\sim\operatorname{Beta}(11,19)\). A posterior predictive replicate is generated by drawing from this posterior and then generating a new count for a trial of the stated size. The replicate is not the observed count copied or re-used: it is a new outcome generated from a parameter distribution informed by the observed count.

Worked Example: Posterior Predictive Replication

Using the observed 9 responses among 20 participants, find the mean and variance of the posterior predictive response count \(Y^{\mathrm{rep}}\) in a new trial of 20 participants, under the same model.

The posterior mean response probability is \(11/(11+19)=11/30=0.3667\). Therefore,

E(Y^{\mathrm{rep}}\mid y_{\mathrm{obs}}) =20\left(\frac{11}{30}\right) =7.333\ \text{responses}.

Using the beta-binomial predictive variance with \(a=11\), \(b=19\), and \(n=20\),

\operatorname{Var}(Y^{\mathrm{rep}}\mid y_{\mathrm{obs}}) =\frac{20(11)(19)(11+19+20)}{(11+19)^2(11+19+1)} =\frac{209000}{27900} =7.491.

The posterior predictive standard deviation is \(\sqrt{7.491}=2.737\) responses. The replicate distribution is centered at 7.333 responses, not at the original prior predictive mean of 4, because the observed count has shifted the parameter distribution. This is a predictive summary, not a claim that the next trial will have exactly 7.333 responses.

Checking a Feature the Model Might Miss

A useful posterior predictive check does not have to focus only on the total response count used to learn the model. It can target a feature of the data that would be concerning if the assumed model could not reproduce it. For example, a model with a common response probability may fit the overall count reasonably while failing to represent marked variation between clinically relevant subgroups.

Worked Example: A Subgroup Discrepancy

Suppose the hypothetical trial enrolled 10 participants in each of two subgroups. The observed counts are 7 responses in subgroup A and 2 in subgroup B, for 9 responses overall. Fit the common-\(p\) model using the total count, giving \(p\mid y_{\mathrm{obs}}\sim\operatorname{Beta}(11,19)\). Define the discrepancy statistic \(T(Y)=|Y_A-Y_B|\), the absolute difference between subgroup response counts.

For the observed data, \(T(y_{\mathrm{obs}})=|7-2|=5\). To check this feature, repeatedly draw \(p\) from \(\operatorname{Beta}(11,19)\), generate two independent \(\operatorname{Binomial}(10,p)\) subgroup counts, and calculate \(T(Y^{\mathrm{rep}})\). For illustration, suppose five such simulated discrepancies are 1, 4, 0, 3, and 3. None is as large as the observed value of 5.

This small set of simulations signals that the observed subgroup contrast merits investigation, but five replicates cannot establish how unusual it is. A larger simulation would estimate the posterior predictive tail area \(P(T(Y^{\mathrm{rep}})\geq 5\mid y_{\mathrm{obs}})\) more reliably. Even then, the check identifies tension between a particular feature of the data and the common-\(p\) model; it does not establish that subgroup membership caused a response difference. The common probability assumption, subgroup definition, chance variation, and data quality all require consideration.

A Short Justification for the Predictive Variance

The beta-binomial variance formula follows from conditioning on the unknown response probability. The law of total variance states that \(\operatorname{Var}(Y)=E[\operatorname{Var}(Y\mid p)]+\operatorname{Var}(E[Y\mid p])\). For \(p\sim\operatorname{Beta}(a,b)\), \(E(p)=a/(a+b)\) and \(\operatorname{Var}(p)=ab/[(a+b)^2(a+b+1)]\). Also, \(E[p(1-p)]=ab/[(a+b)(a+b+1)]\). Substitution gives

\begin{aligned} \operatorname{Var}(Y) &=nE[p(1-p)]+n^2\operatorname{Var}(p)\\ &=\frac{nab}{(a+b)(a+b+1)} +\frac{n^2ab}{(a+b)^2(a+b+1)}\\ &=\frac{nab(a+b+n)}{(a+b)^2(a+b+1)}. \end{aligned}

The two terms have distinct interpretations: the first represents binomial variation conditional on \(p\); the second represents variation induced by uncertainty about \(p\). Ignoring the second term would understate predictive spread whenever the parameter is uncertain.

Interpretation, Diagnostics, and Alternatives

Posterior predictive checks are not formal hypothesis tests with a universal reference distribution. Because the same observed data inform the posterior and define the feature being checked, posterior predictive tail areas can be conservative and should not be interpreted as ordinary frequentist p-values. A large tail area is not proof that the model is correct, and a small one is not an automatic instruction to reject it. The chosen discrepancy, clinical importance, and model assumptions matter.

The approaches answer different questions, as the comparison below summarizes.

Feature	Prior predictive check	Posterior predictive check
Parameter draws	From the prior distribution	From the posterior distribution given observed data
Main question	Does the prior-model combination generate plausible trial outcomes?	Can the fitted model generate data resembling relevant features of the observed trial?
Best timing	Before data, for model and design assessment	After data, for model adequacy assessment
Main limitation	Does not assess fit conditional on the observed results	Can be insensitive to misspecification and is not a calibrated test by default

Neither check replaces other evidence. A likelihood-based goodness-of-fit analysis, a prespecified frequentist diagnostic, or validation on data not used to fit the model may answer a different question. Those alternatives should be selected for the design and purpose; they do not convert a prior or posterior predictive check into a formal test.

Warning signs include simulated values outside the endpoint’s possible range, predictive tails inconsistent with clinical knowledge, subgroup patterns that repeatedly fail to appear in replicated data, or strong sensitivity to an unexamined modeling assumption. Check that the simulation follows the actual design: sample size, randomization structure, follow-up, endpoint definition, and missingness assumptions. A check cannot expose a feature that was neither represented in the data model nor examined by the chosen summary.

Using the wrong parameter distribution. Drawing from the prior after observing the data does not produce a posterior predictive check; drawing from the posterior is required for a check conditioned on the observed data.
Confusing a predictive tail area with the probability the model is true. It is a probability for a defined replicated-data event under a specified model and parameter distribution, not a posterior probability of model truth.
Checking only the overall mean. Similar total response counts can conceal subgroup imbalance or other important patterns. Choose summaries that reflect the clinical and design features of interest.
Treating a few simulated replicates as decisive. A small simulation may make a tail estimate unstable. Report the number of replicates and avoid strong conclusions from a coarse or noisy estimate.

Key takeaway: Prior predictive checks assess what the prior and sampling model imply before using trial results. Posterior predictive checks assess replicated outcomes after the posterior has incorporated those results. State which distribution was used, which feature was checked, and what the check can—and cannot—support.

Check Your Understanding

Answer each question, then compare your reasoning with the solution immediately below it.

A prior predictive simulation draws a response probability from the prior and then generates a trial count. What question does this address?
Solution: It assesses whether the prior together with the sampling model implies clinically plausible trial outcomes before using observed trial results.
In a trial of 20 participants, 8 respond. Under a posterior predictive check, should replicated response counts be generated using prior draws or posterior draws?
Solution: Use posterior draws, because the check is conditioned on the observed data and the posterior distribution incorporates the 8 observed responses.
A posterior predictive simulation gives a small tail area for an observed subgroup discrepancy. Does that prove that subgroup membership causes different response rates?
Solution: No. It indicates tension between that discrepancy and the specified model. It does not establish causation or rule out chance, data-quality problems, or other model limitations.
Why can a posterior predictive check miss a problem even when it is carefully simulated?
Solution: It can miss problems that are not represented by the model or detected by the chosen discrepancy statistic. A check of total responses, for example, may not detect an important subgroup pattern.
What is the central distinction between the two predictive distributions?
Solution: The prior predictive distribution averages the sampling model over the prior; the posterior predictive distribution averages it over the posterior conditional on observed data.