Sensitivity Of Prior Predictive Checks To Prior Parameters

Why Vary Prior Parameters?

A prior predictive check asks what trial outcomes are plausible under a specified model and prior, before observing the trial data. Its conclusions can depend on choices such as the prior mean and the amount of prior concentration around that mean. A single set of prior predictive summaries cannot show which choice is driving an implausible outcome. Sensitivity analysis addresses this by changing prior parameters in a controlled way and comparing the resulting summaries.

The aim is not to find a prior that makes every predicted outcome look comfortable. It is to discover which assumptions matter, how they matter, and whether a clinically important conclusion is robust to reasonable alternatives. A useful comparison changes one feature at a time when possible, states what is held fixed, and interprets changes in the units of the trial outcome.

A Beta-Binomial Model for Prior Sensitivity

Consider a hypothetical trial with a binary endpoint, such as response by a prespecified visit. Let $p$ be the response probability for a participant in the modeled arm, and let $X$ be the number of responses among $n$ participants. Assume that, conditional on $p$, participant outcomes are independent Bernoulli variables with the same response probability $p$. Thus $X\mid p\sim\operatorname{Binomial}(n,p)$. A beta prior, $p\sim\operatorname{Beta}(a,b)$, has positive parameters $a$ and $b$, prior mean $a/(a+b)$, and concentration $k=a+b$.

Definition: A prior predictive distribution for $X$ is the distribution obtained by averaging the sampling distribution of $X$ over the prior distribution of its unknown parameter. In this model, $$ P(X=x)=\int_0^1 P(X=x\mid p)\,f(p)\,dp, $$ where $f(p)$ is the beta prior density. Changing $a$ or $b$ changes this predictive distribution and may change the plausibility assessment of trial outcomes.

The prior mean and concentration describe different features. Holding $a/(a+b)$ fixed while changing $k$ changes how dispersed the prior is around that mean. Holding $k$ fixed while changing $a/(a+b)$ changes the prior’s location. These are distinct sensitivity questions; changing both without explanation makes it harder to identify the source of a predictive change.

For this model, the predictive mean and variance of the number of responses are

E(X)=n\frac{a}{a+b},\qquad \operatorname{Var}(X)=n m(1-m)\frac{n+k}{k+1}, \quad m=\frac{a}{a+b},\quad k=a+b.

The variance includes both the variation in outcomes conditional on $p$ and the variation in $p$ under the prior. Consequently, two priors with the same mean response probability can predict different spreads of trial-level response counts.

Why the Predictive Variance Changes

The variance formula follows from the laws of total expectation and total variance. Conditional on $p$, the binomial model gives $E(X\mid p)=np$ and $\operatorname{Var}(X\mid p)=np(1-p)$. For $p\sim\operatorname{Beta}(a,b)$, its mean is $m=a/k$, and its variance is $m(1-m)/(k+1)$. Also $E[p(1-p)]=m(1-m)-\operatorname{Var}(p)$. Substitution gives

\begin{aligned} \operatorname{Var}(X) &=E\{\operatorname{Var}(X\mid p)\} +\operatorname{Var}\{E(X\mid p)\}\\ &=n\left(m(1-m)-\frac{m(1-m)}{k+1}\right) +n^2\frac{m(1-m)}{k+1}\\ &=n m(1-m)\frac{n+k}{k+1}. \end{aligned}

This derivation shows why the concentration matters. As $k$ grows with $m$ fixed, the prior for $p$ becomes more concentrated and the predictive variance approaches the binomial variance $nm(1-m)$. With smaller $k$, uncertainty about the common response probability adds more variation across replicated trials.

Worked Example: Changing Concentration While Holding the Mean Fixed

Suppose a hypothetical 60-participant trial uses a beta prior for response probability. Compare $\operatorname{Beta}(2,8)$ with $\operatorname{Beta}(20,80)$. Both have prior mean $0.20$, but their concentrations are 10 and 100, respectively. The conditional binomial assumptions are the same for both calculations.

For either prior, the predictive mean is $60(0.20)=12$ responses. For $\operatorname{Beta}(2,8)$, the predictive variance and standard deviation are

\operatorname{Var}(X)=60(0.20)(0.80)\frac{60+10}{10+1} =9.6\frac{70}{11}\approx61.09,\qquad SD(X)=\sqrt{61.09}\approx7.816.

For $\operatorname{Beta}(20,80)$, they are

\operatorname{Var}(X)=60(0.20)(0.80)\frac{60+100}{100+1} =9.6\frac{160}{101}\approx15.21,\qquad SD(X)=\sqrt{15.21}\approx3.900.

The mean is 12 responses under both priors, but the less concentrated prior predicts substantially more variation in the response count across trials. This comparison isolates concentration because the prior mean and trial size are unchanged. The standard deviations describe predictive spread, not a confidence interval for the response probability.

Change the Prior Mean Separately

A second sensitivity question is how outcomes change when the prior mean changes, with concentration held fixed. That comparison can help reveal whether the location of the assumed response rate—not just uncertainty around it—is responsible for a predictive concern. Because the binomial variance depends on $m(1-m)$, changing the mean can also change the predictive spread.

Worked Example: Changing the Prior Mean at Fixed Concentration

For the same hypothetical trial size $n=60$, compare $\operatorname{Beta}(2,18)$ and $\operatorname{Beta}(4,16)$. Both have concentration $k=20$; their prior means are $0.10$ and $0.20$. For the first prior,

E(X)=60(0.10)=6,\qquad \operatorname{Var}(X)=60(0.10)(0.90)\frac{60+20}{20+1} =5.4\frac{80}{21}\approx20.57, \qquad SD(X)\approx4.536.

For the second prior,

E(X)=60(0.20)=12,\qquad \operatorname{Var}(X)=60(0.20)(0.80)\frac{60+20}{20+1} =9.6\frac{80}{21}\approx36.57, \qquad SD(X)\approx6.047.

At the same concentration, moving the prior mean from 0.10 to 0.20 doubles the predictive mean count, from 6 to 12 responses. The standard deviation also increases because $m(1-m)$ is larger at 0.20 than at 0.10. Both changes follow from the model; the increase in spread should not be mistaken for a concentration change.

Prior Sensitivity for Rare Safety Events

A sensitivity check should use summaries that connect to the clinical concern. For a rare safety event, the chance of observing no events may be more informative than the mean alone. This probability depends on the full prior predictive distribution. For the beta-binomial model, integrating the conditional probability $P(X=0\mid p)=(1-p)^n$ over the prior gives $P(X=0)=B(a,b+n)/B(a,b)$, where $B$ is the beta function.

Worked Example: Same Mean Risk, Different Zero-Event Probability

Consider a hypothetical safety assessment with $n=40$ participants and event probability $p$. Compare $\operatorname{Beta}(1,99)$ and $\operatorname{Beta}(2,198)$. Both priors have mean event probability $0.01$, but their concentrations are 100 and 200. Both use the same conditional model: independent participant events given a common $p$.

The predictive mean number of events is $40(0.01)=0.4$ under either prior. For $\operatorname{Beta}(1,99)$, the zero-event probability simplifies to

P(X=0)=\frac{B(1,139)}{B(1,99)}=\frac{99}{139}\approx0.7122.

For $\operatorname{Beta}(2,198)$, it is

P(X=0)=\frac{B(2,238)}{B(2,198)} =\frac{198(199)}{238(239)} =\frac{39402}{56882}\approx0.6927.

Thus, despite the same prior mean event risk and the same expected event count, the predicted chance of seeing no events differs by about 0.0195, or 1.95 percentage points. Whether that difference matters depends on the clinical question and the decision context; the calculation identifies an assumption-sensitive summary rather than declaring either prior acceptable.

A Practical Sensitivity Workflow

A controlled comparison makes it easier to explain which assumption drives a change. First state the model, trial size, and clinical summaries of interest. Then specify the prior parameter being perturbed and what is kept fixed. Recalculate the same summaries for each perturbation and interpret differences in context. A small grid of plausible parameter values can reveal whether the response is smooth or whether a modest change crosses a clinically meaningful boundary.

Choose the clinical summary.
Examples include the expected number of responses, the predictive spread, a probability of zero safety events, or a specified tail probability.

Identify the assumption to vary.
For a beta prior, vary the mean while holding concentration fixed, or vary concentration while holding the mean fixed. State the fixed quantity explicitly.

Compare on a common scale.
Use the same trial size, outcome definition, and summary for every parameter setting; report count units or probability differences as appropriate.

Interpret the change and its scope.
Explain which parameter change produced the difference, whether it affects a clinical benchmark, and what the comparison does not establish.

One-at-a-time perturbations are useful for identifying the influence of an individual parameter, but they do not cover every plausible combination. If mean and concentration are both uncertain, separate analyses may be followed by checks of selected joint settings. This is different from an exhaustive ranking of candidate priors: the immediate question here is which parameter assumptions drive the chosen predictive summaries.

Diagnostics, Alternatives, and Limitations

A sensitivity analysis is informative only if the model and calculations remain coherent across the settings compared. Check that all beta parameters are positive, that the stated means and concentrations match $a$ and $b$, and that trial size and endpoint definition have not changed between calculations. For simulated summaries, also check that apparent differences are larger than ordinary simulation noise; simulation error is distinct from sensitivity to the prior itself.

If a result changes sharply under a small, clinically plausible parameter perturbation, report that dependence rather than presenting only the preferred setting. If a result barely changes across the examined range, that is evidence of robustness only over that range and for those summaries. It does not establish that the model is correct, that unexamined prior features are harmless, or that the trial will behave as predicted.

Analytic summaries, such as the beta-binomial mean, variance, and zero-event probability, are efficient when their model applies. Simulation can instead estimate summaries that lack a convenient closed form or represent a more complex trial design. Graphical comparisons can make changes across a parameter grid easier to see, while clinical benchmarks help determine whether a numerical difference matters in practice. These approaches answer complementary questions; none removes the need to state the model and assumptions.

The beta-binomial illustration assumes a common event probability across participants and conditional independence given that probability. If risks differ systematically by subgroup, outcomes are dependent for reasons not represented by this model, or follow-up varies, these assumptions may be inadequate. A change in predictive spread can then reflect model structure as well as prior parameters. Sensitivity to prior parameters cannot diagnose every structural misspecification.

Common Pitfall

Common mistake: “Both priors have the same mean, so their prior predictive distributions are the same.”

Correction: The mean alone does not determine predictive spread or tail probabilities. In the beta-binomial model, priors with a common mean but different concentrations have the same $E(X)$, but generally different $\operatorname{Var}(X)$ and different probabilities for particular trial outcomes. Compare the summary relevant to the clinical question, not only the mean.

Another easy-to-miss error is changing both the mean and concentration and then attributing the entire predictive difference to one of them. Keep one fixed when the goal is to isolate the effect of the other, and describe any joint changes as joint changes.

Key takeaway: Prior predictive sensitivity is a controlled investigation of how prior assumptions affect predicted trial outcomes. Separating prior location from concentration, comparing the same clinically relevant summaries, and stating the model’s limits make the results interpretable.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For $n=50$, a beta prior has mean $0.20$ and concentration 20. What is the prior predictive mean response count?
In a beta-binomial model, what should be held fixed to assess sensitivity to concentration rather than prior mean?
For $n=40$, compare the predictive variances under $m=0.10,k=20$ and $m=0.20,k=20$. Which is larger?
Two priors give the same expected safety-event count but different probabilities of zero events. Is that contradictory? Explain.

Solutions. (1) $E(X)=nm=50(0.20)=10$ responses. (2) Hold the prior mean $m=a/(a+b)$ fixed and vary $k=a+b$, with the trial design unchanged. (3) The variances are $40(0.10)(0.90)(60/21)\approx10.29$ and $40(0.20)(0.80)(60/21)\approx18.29$ responses squared; the second is larger. (4) No. The expected count does not determine the full predictive distribution, so priors with the same mean event risk can assign different probabilities to zero events.