Prior Predictive Review of a Rare-Disease Trial

Why Review a Prior Predictively?

A prior distribution can look mathematically convenient and still imply trial outcomes that are difficult to reconcile with clinical knowledge. In a small rare-disease trial, a prior on a response probability may put substantial probability on zero responses, or may predict many more responses than are clinically credible. A treatment prior may also imply effects so large that they deserve scrutiny before any trial data are observed.

A prior predictive review translates assumptions about unknown response probabilities into distributions for observable quantities, such as the number of responses among the planned participants. This makes the assumptions easier to discuss with clinicians and regulators: a response count has a direct trial-scale interpretation, whereas a prior parameter may not.

Definition: Suppose the response probability $p$ has prior distribution $\pi(p)$, and the response count $K$ has sampling distribution $P(K\mid p)$. The prior predictive distribution of $K$ is $$ P(K=k)=\int P(K=k\mid p)\,\pi(p)\,dp. $$ It describes the distribution of possible response counts before observing the trial outcomes, under the specified prior and sampling model.

For a single arm with $n$ participants, this review commonly assumes a fixed sample size, a binary response measured over a specified assessment window, and conditionally independent responses with a common probability $p$. The common probability is uncertain and given a prior distribution. These assumptions matter: unequal follow-up, clustering, or meaningful variation in response probabilities across participants may require a different sampling model.

Beta-Binomial Prior Prediction

For a binary endpoint, let $K$ be the number of responses among $n$ participants. Use $K\mid p\sim\operatorname{Binomial}(n,p)$ and $p\sim\operatorname{Beta}(a,b)$, with $a,b>0$. Mixing the binomial sampling distribution over the beta prior gives the beta-binomial distribution.

P(K=k) =\binom{n}{k}\frac{B(a+k,b+n-k)}{B(a,b)}, \qquad k=0,1,\ldots,n.

Here $B(\cdot,\cdot)$ is the beta function. The prior mean response probability is $\mu=E(p)=a/(a+b)$, so the predictive mean response count is $E(K)=n\mu$. The predictive variance includes both variation in outcomes at a fixed $p$ and uncertainty about $p$:

\begin{aligned} \operatorname{Var}(K) &=E\{\operatorname{Var}(K\mid p)\} +\operatorname{Var}\{E(K\mid p)\}\\ &=E\{np(1-p)\}+\operatorname{Var}(np)\\ &=\frac{nab}{(a+b)(a+b+1)} +\frac{n^2ab}{(a+b)^2(a+b+1)}\\ &=\frac{nab(a+b+n)}{(a+b)^2(a+b+1)}. \end{aligned}

The first term is $E\{np(1-p)\}$, not $n\mu(1-\mu)$. The latter treats $p$ as fixed at its prior mean and omits uncertainty in $p$. The second term is $\operatorname{Var}(np)$. This decomposition explains why the beta-binomial count distribution is generally more variable than a binomial distribution evaluated at the prior mean.

Worked Example: Predicting Control Responses Under a Low-Rate Prior

Suppose a hypothetical rare-disease trial plans to observe $n=20$ participants in its control arm. A proposed prior is $p_C\sim\operatorname{Beta}(1,19)$. Assume each participant has the same binary response definition and assessment window, and responses are conditionally independent given $p_C$.

The prior mean control response probability is $1/(1+19)=0.05$, or 5%. The prior predictive mean count is therefore $$ E(K_C)=20(0.05)=1.00\text{ response}. $$

The probability of no control responses is $$ P(K_C=0)=\frac{B(1,39)}{B(1,19)} =\frac{19}{39}=0.4872. $$ Thus, despite a prior mean of one response, nearly half of the prior predictive probability is on observing no responses in this 20-participant arm. This may be plausible for a low-rate endpoint, but it should be assessed against the clinical setting rather than assumed to be acceptable.

For the variance calculation, $a+b=20$ and $ab=19$. The two law-of-total-variance terms are $$ E\{20p_C(1-p_C)\}=\frac{20(19)}{20(21)}=0.9048, \qquad \operatorname{Var}(20p_C)=\frac{20^2(19)}{20^2(21)}=0.9048. $$ Their sum is $\operatorname{Var}(K_C)=1.8095$, with predictive standard deviation $\sqrt{1.8095}=1.345$ responses. The variability is larger than the binomial variance $20(0.05)(0.95)=0.9500$, which would ignore prior uncertainty in the response probability.

Reviewing a Treatment-Arm Prior

For a randomized two-arm trial, let $p_T$ and $p_C$ be the treatment and control response probabilities. A direct prior predictive review generates counts $K_T$ and $K_C$ from the planned arm sizes under the proposed priors. If the arm-specific probabilities have independent priors and responses are conditionally independent within and between arms, then the marginal prior predictive counts are independent. If the prior couples the arm probabilities, their predictive counts need not be independent.

Useful summaries include the probability of no responses, the chance of counts above a clinically plausible range, and the prior predictive distribution of a contrast such as $\Delta=p_T-p_C$. For treatment effects, review the implied probabilities of negligible benefit, clinically important benefit, and implausibly large benefit. These are prior probabilities, not probabilities calculated from trial outcomes.

Worked Example: Predicting Treatment Responses and a High Rate

Suppose a hypothetical treatment arm has $n_T=10$ participants and prior $p_T\sim\operatorname{Beta}(2,8)$. Assume a common response probability and conditionally independent binary outcomes. Its prior mean response probability is $2/10=0.20$, so the predictive mean is $$ E(K_T)=10(0.20)=2.00\text{ responses}. $$

The probability of zero responses is $$ P(K_T=0)=\frac{B(2,18)}{B(2,8)} =\frac{1/(18\cdot19)}{1/(8\cdot9)} =\frac{72}{342}=0.2105. $$ Thus the prior predicts no response in about 21.1% of such hypothetical arms, even though its mean count is two.

The prior predictive variance is $$ \operatorname{Var}(K_T) =\frac{10(2)(8)(10+10)}{10^2(11)} =2.909, $$ so its predictive standard deviation is 1.706 responses. Another useful check is the prior probability that the response probability exceeds 0.15. For a $\operatorname{Beta}(2,8)$ prior, $$ P(p_T>0.15)=(1-0.15)^8\{1+8(0.15)\} =0.85^8(2.2)=0.5995. $$ Nearly 60.0% prior probability is above a 15% response rate. Whether that is reasonable depends on the disease, endpoint, and treatment mechanism; it is not established by the prior mean alone.

Checking Clinical Plausibility and Treatment Effects

A prior predictive review should ask whether the implied observations and effects make sense together. A treatment prior with a plausible mean can still have substantial tails at rates considered unrealistic. Conversely, a prior that assigns little probability to extreme rates may still produce a wide range of small-trial counts because even a fixed response probability produces sampling variation.

For a treatment-effect check, the event of interest must be stated precisely. For example, $p_T-p_C>0.30$ means that the treatment response probability exceeds the control response probability by more than 30 percentage points. The probability of this event is computed under the joint prior for $p_T$ and $p_C$; it cannot generally be inferred by subtracting their means or by inspecting their marginal priors separately.

Worked Example: Detecting an Implausibly Broad Treatment Prior

Consider a hypothetical trial with 12 participants per arm. Suppose the control prior is $p_C\sim\operatorname{Beta}(1,19)$, while the treatment prior is $p_T\sim\operatorname{Beta}(1,1)$, independently. Assume the same binary endpoint and assessment window in both arms, with conditionally independent responses.

The control prior mean is 0.05 and the control predictive mean is $12(0.05)=0.60$ responses. Its probability of no responses is $$ P(K_C=0)=\frac{B(1,31)}{B(1,19)}=\frac{19}{31}=0.6129. $$ For treatment, the uniform $\operatorname{Beta}(1,1)$ prior gives a uniform beta-binomial predictive distribution: $$ P(K_T=k)=\binom{12}{k}\frac{B(k+1,13-k)}{B(1,1)} =\frac{12!}{k!(12-k)!}\frac{k!(12-k)!}{13!} =\frac{1}{13}. $$ Consequently, $E(K_T)=6.00$ responses, and the prior assigns probability $1/2$ to $p_T>0.50$. If response rates above 50% are not credible for this treatment and endpoint, this prior has an obvious clinical-plausibility problem.

The prior also assigns probability about 0.65 to a treatment advantage greater than 30 percentage points. Conditional on $p_C=c$, the uniform treatment prior gives $P(p_T-c>0.30\mid p_C=c)=\max(0,0.70-c)$. Therefore, $$ \begin{aligned} P(p_T-p_C>0.30) &=\int_0^{0.70}(0.70-c)19(1-c)^{18}\,dc\\ &=0.70-E(p_C)+\int_{0.70}^1 P(p_C>t)\,dt\\ &=0.65+\frac{0.30^{20}}{20}\\ &\approx 0.6500. \end{aligned} $$ The final tail term is approximately $1.743\times10^{-12}$. The calculation makes explicit that the broad treatment prior implies a very high probability of a large treatment advantage. This is a reason to revisit the prior assumptions, not evidence that such an effect will occur.

How to Conduct the Review

State the estimand and sampling model.
Specify the response definition, assessment window, arm sizes, and assumptions about the response probabilities and participant outcomes.

Generate prior predictive quantities.
Calculate count distributions analytically when possible; otherwise simulate probabilities from the prior and outcomes from the sampling model.

Compare with clinical constraints.
Review prior rates, counts, and treatment-effect contrasts against plausible ranges informed by the disease and endpoint.

Diagnose and document concerns.
Investigate surprising tails, revise assumptions if justified, and record the rationale and remaining uncertainty.

When simulation is used, draw $p_T$ and $p_C$ from their joint prior, then draw the arm counts from their binomial distributions conditional on those probabilities. Summaries such as quantiles, probabilities of zero responses, and probabilities of a clinically important contrast should be stable to increased simulation size. Check that simulated probabilities lie in $[0,1]$, that counts lie between zero and the planned arm size, and that simulated means agree with analytic expectations when available.

A prior predictive review is different from a posterior predictive check: it uses the prior before observing outcomes, whereas a posterior predictive check conditions on the observed data through the posterior. It is also distinct from sensitivity analysis, which compares conclusions across alternative prior specifications. Both can be useful, but neither replaces examining what a proposed prior predicts before data are seen.

Alternatives, Limitations, and Common Pitfalls

For a beta-binomial model, exact calculations are often clearest for a single arm, while simulation is flexible for joint treatment effects, complex designs, or nonconjugate priors. A scenario table based on selected response probabilities can help communicate sampling variation at fixed rates, but it does not represent uncertainty in $p$ unless probabilities are weighted according to a prior. These approaches answer related but different questions.

Prior predictive plausibility is not a mechanical rule for selecting a prior. Clinical knowledge may be limited, and several priors may produce plausible observable outcomes while differing in how they distribute probability across rates. The review cannot establish that a model is correct, nor can it fix a misspecified endpoint, an incorrect follow-up model, or unmodeled heterogeneity. If outcomes are dependent, the binomial assumption can understate or misrepresent count variability. If response probabilities differ meaningfully across participants or sites, a shared-$p$ model may be inadequate.

Pitfall: checking only the prior mean. A mean control rate of 5% does not tell you how likely zero responses are in a small arm. Check predictive counts and tail probabilities as well.
Pitfall: treating a count distribution as binomial at the prior mean. This omits uncertainty in $p$. Integrate over the prior or use the beta-binomial distribution when its assumptions hold.
Pitfall: judging treatment and control priors separately. A treatment-effect claim depends on their joint distribution. Calculate the contrast probability under that joint prior.
Pitfall: calling a prior predictive probability a clinical fact. It describes implications of the model and prior; it does not validate either one.

Key takeaway: Before interpreting trial data, translate each proposed response-rate prior into predicted event counts and examine the joint prior implications for treatment effects. A clinically credible prior should be defensible not only by its parameters, but also by the trial outcomes it makes plausible.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

With $p\sim\operatorname{Beta}(1,19)$ and $n=10$, what are the prior predictive mean count and probability of zero responses?
For $p\sim\operatorname{Beta}(2,8)$, calculate the prior mean and $P(p>0.15)$.
For $p\sim\operatorname{Beta}(1,1)$ and $n=5$, what is the probability of each possible response count?
How does a prior predictive check differ from a posterior predictive check?
Why can a shared response probability be inappropriate if participants have substantially different response probabilities?

Solutions. (1) The mean is $10(1/20)=0.5000$ responses. The zero-count probability is $B(1,29)/B(1,19)=19/29=0.6552$. (2) The prior mean is $2/(2+8)=0.20$. The tail probability is $0.85^8\{1+8(0.15)\}=0.5995$. (3) The beta-binomial distribution is uniform, so $P(K=k)=1/6$ for each $k=0,1,\ldots,5$. (4) A prior predictive check averages the sampling model over the prior before observing outcomes; a posterior predictive check uses the posterior after conditioning on observed data. (5) Conditional independence with one common probability may fail, and the resulting model can misrepresent the variability and distribution of total responses.