Prior Predictive Checks for Decision Rules

Why Check the Prior Predictive Distribution?

A prior distribution can look reasonable when considered parameter by parameter yet imply implausible trial data or decision behavior. For example, a prior for response rates may put substantial probability on nearly universal response, or a prior for treatment effects may imply that a decision rule selects an arm unusually often before any trial data are observed. A prior predictive check makes these implications visible by asking what outcomes and decisions the model would generate before observing the trial.

In a multi-arm trial, this check should reflect the full structure used for the decision: the outcome model, the priors on control and treatment parameters, any dependence induced by a shared control, and the actual posterior probability threshold. It is not a substitute for examining posterior decisions after data arrive. It is a pre-data check of the assumptions that feed those decisions.

Definition: Let $\theta$ denote the unknown model parameters and $Y$ the trial data. The prior predictive distribution is $$ p(y)=\int p(y\mid\theta)\,\pi(\theta)\,d\theta, $$ where $p(y\mid\theta)$ is the sampling model and $\pi(\theta)$ is the prior distribution. A prior predictive check examines summaries or events generated from this distribution, such as outcome rates, treatment contrasts, or whether a specified decision rule would act.

The check has two stages: draw parameter values from the prior, then draw trial data conditionally from the sampling model. For each simulated data set, one may calculate the relevant posterior probability and apply the prespecified decision rule. The resulting frequency of decisions is a prior predictive decision probability. It averages over the prior and the sampling model; it is not the probability of success under a single fixed true effect, nor a frequentist type I error rate.

Model the Outcomes Before Checking Them

A prior predictive calculation is only as meaningful as its model. State the sampling assumptions and prior explicitly, and make sure the prior is on parameters with an interpretable scale. For a binary outcome, a common arm-level model is $X_j\mid p_j\sim\operatorname{Binomial}(n_j,p_j)$, where $X_j$ is the number of responses in arm $j$ and $p_j$ is its response probability. If $p_j\sim\operatorname{Beta}(a,b)$, integrating out $p_j$ gives a beta-binomial prior predictive distribution.

P(X_j=x)=\binom{n_j}{x}\frac{B(a+x,b+n_j-x)}{B(a,b)},\qquad x=0,\ldots,n_j.

Its mean is $n_j a/(a+b)$, but a check should not stop at the mean. The full distribution shows how much prior predictive mass lies on very low or high response counts. In a multi-arm setting, one should also inspect treatment-control contrasts or joint events across arms; marginally plausible rates can still imply implausible differences or an unexpected frequency of decisions.

Worked Example: A Normal Prior and the Chance of Selecting an Arm

Consider one experimental arm and a control arm with continuous outcomes. Assume independent normal observations with known standard deviation $1$, and $16$ participants per arm. Let $\Delta$ be the treatment-minus-control mean difference, with positive values favoring treatment. The observed difference $Z=\bar{x}_1-\bar{x}_0$ then satisfies $Z\mid\Delta\sim N(\Delta,0.125)$, since $1/16+1/16=0.125$. Suppose the prior is $\Delta\sim N(0,0.25)$, and the arm is selected if $P(\Delta>0.2\mid Z)>0.90$. These assumptions specify both the outcome model and the prior; the calculation does not account for uncertainty in the known standard deviation.

By normal-normal conjugacy, the posterior variance and mean are

V=\left(\frac{1}{0.25}+\frac{1}{0.125}\right)^{-1}=\frac{1}{12},\qquad m(Z)=V\frac{Z}{0.125}=\frac{2}{3}Z.

The posterior probability rule is equivalent to a threshold on $Z$. Under the posterior normal distribution, $P(\Delta>0.2\mid Z)>0.90$ exactly when

\frac{m(Z)-0.2}{\sqrt{1/12}}> \Phi^{-1}(0.90)=1.2816.

Thus $m(Z)>0.2+1.2816\sqrt{1/12}\approx0.5700$, so $Z>0.5700/(2/3)\approx0.8550$ outcome units. Before observing data, the prior predictive distribution is $Z\sim N(0,0.25+0.125)=N(0,0.375)$. Therefore the prior predictive probability that this rule selects the arm is

P(Z>0.8550)=1-\Phi\left(\frac{0.8550}{\sqrt{0.375}}\right) \approx 1-\Phi(1.396)\approx0.081.

Under this prior and model, the rule selects the arm in about 8.1% of hypothetical pre-data trials. Whether that frequency is acceptable depends on the clinical context and the intended role of the rule. It is not a claim that the treatment has an 8.1% chance of being effective.

Checking Binary-Outcome Rules

For a binary outcome, it is useful to connect a posterior threshold to counts that would trigger a decision. This reveals whether a rule is driven by data or effectively predetermined by the prior. The following example uses a minimum activity criterion for each experimental arm. The criterion is an absolute response rate, not a comparison with the shared control; a comparative rule would require the joint prior and likelihood for treatment and control.

Worked Example: How Often Does a Beta-Binomial Rule Select?

Suppose each experimental arm has $n=10$ participants with independent binary outcomes, and $p\sim\operatorname{Beta}(1,1)$. An arm is selected if its posterior probability of exceeding a minimum response rate of $0.20$ is greater than $0.90$. These assumptions mean the response count is conditionally binomial and the prior is uniform on the response probability.

If $x$ responses are observed, the posterior is $\operatorname{Beta}(x+1,11-x)$. For integer beta parameters, the identity $P\{\operatorname{Beta}(a,b)>c\}=P\{\operatorname{Binomial}(a+b-1,c)\leq a-1\}$ gives the needed tail probabilities. At $x=3$, the posterior is $\operatorname{Beta}(4,8)$, so

P(p>0.20\mid x=3)=P\{\operatorname{Binomial}(11,0.20)\leq3\} =0.8388608.

This does not exceed $0.90$, so the rule does not select the arm. At $x=4$, the posterior is $\operatorname{Beta}(5,7)$, giving

P(p>0.20\mid x=4)=P\{\operatorname{Binomial}(11,0.20)\leq4\} =0.9495904.

The rule selects at four responses, and the posterior probability increases with $x$, so it selects exactly when $x\geq4$. Under the uniform prior, the beta-binomial prior predictive distribution for $X$ is uniform on $0,\ldots,10$: each count has probability $1/11$. Consequently, the prior predictive probability of selection is

P(X\geq4)=\frac{7}{11}\approx0.6364.

For two arms with independent priors and independent outcomes, each using this same rule, the prior predictive probability that both are selected is $(7/11)^2\approx0.4050$, and the probability that at least one is selected is $1-(4/11)^2=105/121\approx0.8678$. Those calculations rely on independence; they would not generally hold with shared uncertain parameters or other dependence.

The beta-binomial identity also provides a useful arithmetic check: for $x=3$ and $x=4$, the binomial probabilities are respectively $P\{\operatorname{Binomial}(11,0.20)\leq3\}$ and $P\{\operatorname{Binomial}(11,0.20)\leq4\}$. Small errors in these tails can change whether a count crosses a strict decision threshold, so retain sufficient precision until the final comparison.

Shared Controls and Joint Prior Predictive Checks

In a multi-arm trial, observed treatment-control contrasts are dependent when they use the same control participants, even if the experimental-arm samples are independent. A prior predictive check that simulates arms separately and then treats the resulting contrasts as independent can therefore misrepresent how often several arms appear extreme together.

Worked Example: Dependence Between Two Shared-Control Contrasts

Suppose outcomes are normal with known standard deviation $1$, and $25$ participants are assigned to each of two experimental arms and one shared control. Let the true effects $\Delta_1,\Delta_2$ be independent $N(0,0.09)$. Write the observed contrasts as $D_j=\bar{x}_j-\bar{x}_0$. This model assumes independent participant outcomes across the three groups and independent effect priors.

Each sample mean has variance $1/25=0.04$. Hence

\operatorname{Var}(D_j)=0.09+0.04+0.04=0.17,\qquad \operatorname{Cov}(D_1,D_2)=\operatorname{Var}(\bar{x}_0)=0.04.

The covariance is positive because the same control mean is subtracted from both experimental means. The prior predictive correlation is $0.04/0.17\approx0.2353$. For either arm individually, the prior predictive probability that the observed contrast has magnitude greater than $1$ outcome unit is

P(|D_j|>1)=2\Phi\left(-\frac{1}{\sqrt{0.17}}\right) \approx 2\Phi(-2.425)\approx0.0153.

That marginal probability does not determine the chance that both contrasts exceed the threshold. Because the contrasts are correlated, multiplying $0.0153$ by itself would incorrectly assume independence. A joint check should simulate the shared control observation once per trial replicate, then use it in both contrasts.

A Practical Checking Workflow

Prior predictive checks are most informative when they connect scientific judgments about plausible outcomes to the decisions that will actually be used. A practical sequence is:

Specify the joint model.
State the sampling model, prior distribution, arm structure, and any shared-control or hierarchical dependence. Check that the assumed likelihood is appropriate for the endpoint and design.

Choose interpretable summaries.
Inspect outcome rates, event counts, treatment-control contrasts, and joint events relevant to the trial. Include clinically implausible extremes rather than checking only prior means.

Apply the actual rule.
For simulated data, calculate the posterior probability specified by the planned rule and record which arms would be selected. Preserve the trial’s shared-control and multi-arm structure in every replicate.

Review and revise transparently.
Compare the implied outcomes and decision frequencies with clinical knowledge and design intent. If a prior is revised, document the reason and repeat the checks under the revised assumptions.

Analytic calculations are useful for simple conjugate models; simulation is usually more flexible for hierarchical priors, correlated parameters, and complex rules. In simulation, draw $\theta^{(r)}$ from the joint prior, draw $Y^{(r)}$ from the trial likelihood conditional on that draw, and apply the rule to $Y^{(r)}$. The estimated prior predictive probability of an event is the proportion of replicates in which it occurs. Monte Carlo error matters: for an estimated probability $\hat q$ based on $R$ independent replicates, its approximate standard error is $\sqrt{\hat q(1-\hat q)/R}$. Increase $R$ when rare events or close comparisons matter.

What the Checks Can and Cannot Tell You

A prior predictive check can expose a prior that places substantial probability on impossible or implausible response rates, effects, or trial-level outcomes. It can also show whether a decision rule tends to select many arms, no arms, or particular combinations under the model’s prior beliefs. In that way, it complements sensitivity analysis: sensitivity analysis compares decisions across alternative priors, while prior predictive checking asks what each prior implies for observable data and decisions.

The check does not establish that a prior is objectively correct. It cannot rule out likelihood misspecification, and plausibility judgments must come from relevant clinical and scientific knowledge. Nor is the prior predictive decision probability a frequentist operating characteristic at a fixed parameter value. If repeated-sampling error rates or performance under specified true effects are needed, those require calculations under those fixed data-generating values.

A further distinction is from posterior predictive checking. Prior predictive checks use $\pi(\theta)$ before the trial data are observed. Posterior predictive checks use the posterior distribution after observing data and assess whether the fitted model can reproduce features of those data. The two checks answer different questions; neither can replace the other.

Common mistake: “The prior predictive rule selects an arm in 64% of simulated trials, so the treatment has a 64% chance of being effective.” This confuses a model-averaged frequency of a decision with the probability that a treatment is truly effective.
Correction: Report the quantity precisely: under the specified prior, likelihood, and rule, the rule selects in about 64% of prior predictive trials. Separately report posterior probabilities after observing data, or performance under fixed assumed treatment effects when that is the target.
Diagnostic warning: If simulated rates or contrasts are implausible, inspect prior scales, dependence assumptions, and outcome-model constraints. Do not respond by changing the decision threshold without understanding which model implication caused concern.

Key takeaway: A prior predictive check evaluates the observable and decision-level consequences of the prior combined with the likelihood. Preserve multi-arm dependence, use the planned posterior rule, and interpret the resulting frequency as prior predictive—not as posterior evidence or a fixed-truth error rate.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In one sentence, distinguish a prior predictive check from a posterior predictive check.
Under the beta-binomial example, what fraction of prior predictive data sets lead to selection for one arm?
Why is it incorrect to multiply the marginal probabilities of two shared-control contrasts exceeding a threshold when calculating the chance that both exceed it?
A prior predictive simulation produces a high frequency of arm selection. Does this show that the treatment is likely effective? Explain.

Solutions. (1) A prior predictive check draws parameters from the prior and data from the likelihood before observing trial data, whereas a posterior predictive check draws parameters from the posterior after observing data. (2) Selection occurs at counts $X\geq4$, so the probability is $7/11\approx0.6364$. (3) Both contrasts contain the same random control mean, giving covariance $0.04$ in the worked model; the events are therefore dependent, and their joint probability is not generally the product of their marginal probabilities. (4) No. The frequency describes how often the specified rule acts across trials generated from the prior and likelihood; it is not the probability that a treatment is effective.