Comparing Candidate Priors Through Generated Outcomes

Why Compare Priors Through Generated Outcomes?

Two candidate priors can look similar in parameter space yet imply quite different outcomes for a planned trial. Conversely, priors with visibly different parameter spreads may lead to outcomes that are practically alike at the planned sample size. Comparing generated outcomes puts the question in trial terms: what event counts, treatment differences, and unusual results would this design produce if each candidate prior were used?

The comparison is meaningful only when the trial design and outcome model are held fixed. Otherwise, a difference in generated results could arise from a changed sample size, allocation, follow-up, or likelihood rather than from the priors. The goal here is not to select a prior automatically, but to make the consequences of candidate priors visible and judge them against clinical knowledge.

A Fair Comparison: Fix the Trial, Vary the Prior

Suppose a planned two-arm trial assigns 100 participants to control and 100 to treatment, and records whether each participant experiences a binary event by a specified follow-up time. Let \(p_C\) and \(p_T\) denote the event probabilities in the control and treatment arms. Conditional on these probabilities, assume independent binomial outcomes:

This model assumes a common event probability within each arm, independent participant outcomes conditional on the probabilities, complete outcome ascertainment, and the stated fixed sample sizes. Candidate priors may express different beliefs about the probabilities or their dependence. For the examples below, arm-specific priors are independent. That independence is an assumption, not a requirement: if prior knowledge links the arm probabilities, the joint prior must represent that dependence.

For a direct comparison, use the same endpoint definition, follow-up, allocation, sample sizes, likelihood, and summary measures for every candidate. A useful generated replicate proceeds by drawing the arm probabilities from a candidate prior and then drawing arm counts from the corresponding binomial distributions. Repeating this produces a distribution of possible trial results for that candidate.

What the Prior Does to Event-Count Variation

A simple calculation helps interpret simulated count distributions. If \(p\sim\operatorname{Beta}(a,b)\) and \(Y\mid p\sim\operatorname{Binomial}(n,p)\), then \(Y\) has a beta-binomial prior predictive distribution. Write \(s=a+b\) and \(\mu=a/(a+b)\). Its mean and variance are:

The variance contains both variation in binomial outcomes conditional on \(p\) and variation in \(p\) across prior draws. The following justification uses the law of total expectation and variance. Since \(E(Y\mid p)=np\) and \(\operatorname{Var}(Y\mid p)=np(1-p)\), and a beta variable has \(E(p)=\mu\) and \(\operatorname{Var}(p)=\mu(1-\mu)/(s+1)\), the mean is \(E\{E(Y\mid p)\}=n\mu\). For the variance:

At a fixed prior mean, smaller \(s\) means a more diffuse beta prior and generally a broader predictive count distribution. The binomial sampling variation remains even if the prior becomes very concentrated; prior uncertainty adds variation beyond that conditional sampling component.

Compare Treatment Differences as Well as Arm Counts

A clinically important comparison often concerns the treatment-minus-control event-count difference \(D=Y_T-Y_C\), or the corresponding difference in event rates \(D/100\). Looking at each arm separately can miss whether the candidate prior implies implausibly large benefits or harms. Under independent arm priors and independent arm outcomes, the predictive means of the counts subtract, while their predictive variances add.

Read Generated Outcomes in Clinical Terms

Simulation makes distributional features easier to inspect than a few moments alone. For example, suppose an analyst generates 20,000 hypothetical trials under each candidate in the preceding examples. The table below gives an illustrative set of simulation summaries; the frequencies are hypothetical outputs, not exact probabilities. The event-count threshold and treatment-difference threshold would need clinical justification in an actual application.

A frequency such as 13.0% estimates the probability of that event under Candidate A's prior predictive distribution for the specified design. It is not a probability that the prior is correct, nor does it establish that the threshold is clinically unacceptable. The appropriate interpretation depends on the trial context: an event count of 20 may be plausible in a large safety trial but concerning in another setting.

Diagnostics, Alternatives, and Limitations

Generated-outcome comparisons are only as useful as their shared model and summaries. Check whether the model represents the design and clinical endpoint. If the endpoint involves clustering, repeated measurements, competing events, or a different follow-up structure, a simple independent binomial model may not be adequate. Also inspect whether prior draws remain in valid parameter ranges and whether any dependence between baseline risk and treatment effect has been included when clinically justified.

Compare more than the center of a distribution. Useful summaries include event-count quantiles, treatment-difference quantiles, and prespecified probabilities of clinically important or extreme outcomes. Plotting the distributions against clinical benchmarks can show whether a small tail probability conceals an implausible central region, or vice versa. The earlier material on prior predictive quantiles, tail behavior, and graphical clinical benchmarks provides related tools for these summaries.

A common mistake is to conclude, “The two priors have the same mean event rate, so they imply the same trial outcomes.” Equal means do not imply equal predictive spreads or tail probabilities, as the beta-binomial calculations show. The correction is to compare the generated distributions, not only their centers. A second pitfall is to change the sample size or likelihood for one candidate; that no longer isolates the prior's implications.

Analytic predictive calculations, such as the beta-binomial moments above, are valuable checks on simulation summaries when they are available. Simulation is more flexible for complex models and joint outcomes, but its summaries are approximations based on generated replicates. Neither approach determines which prior is scientifically appropriate: that judgment requires substantive knowledge and transparent clinical benchmarks. A prior predictive comparison also does not use observed trial outcomes and is not a posterior analysis.