Prior Predictive Checks For Binary Overdispersion

Why Check Binary Overdispersion?

For a binary clinical endpoint, a binomial model is often a useful starting point: among \(n\) patients, each has the same event probability \(p\), and outcomes are conditionally independent. But a model that allows shared or correlated risk can generate trial event counts with greater variability than this binomial benchmark. Such extra variability is called overdispersion relative to the binomial model.

Prior predictive checks ask what outcomes the proposed model and prior can generate before using trial data to update beliefs. For binary overdispersion, replicate whole trials from the prior predictive distribution and examine event counts or event rates across replicates. The question is not simply whether counts vary—they must vary by chance—but whether their variability, clustering, or extreme values are clinically plausible.

The key modeling distinction is whether patient risks vary independently or share a trial-level source of variation. Independent patient-specific risks do not necessarily overdisperse the total count. A shared risk drawn once per trial induces dependence among that trial's outcomes and can increase count variance. The hierarchy used to generate replicates therefore matters as much as the marginal mean risk.

A Shared Trial Risk and the Beta-Binomial Model

Here \(P_j\) is drawn once for a trial, then shared by its patients. Across replicated trials, the \(P_j\) values are independently drawn from the same beta distribution. The beta mean is \(\mu_P=a/(a+b)\), and its variance is \(ab/[(a+b)^2(a+b+1)]\). Write \(\rho=1/(a+b+1)\). This parameter equals the correlation between two different outcomes in the same trial under the model.

The binomial model with the same mean event probability has variance \(n\mu_P(1-\mu_P)\). Therefore, when \(n>1\) and the beta distribution has nonzero variance, \(\rho>0\) and the beta-binomial count has larger variance than that fixed-probability binomial model. For \(n=1\), the multiplier is one: a single outcome is Bernoulli with probability \(\mu_P\), so the two count distributions are identical. A prior on the probability cannot create additional variation in a one-patient count.

Why the Variance Increases

The result follows from the law of total variance: the variability in \(K_j\) consists of the average binomial variability conditional on \(P_j\), plus variability in the conditional mean \(nP_j\) across trials.

For the last equality, \(\operatorname{Var}(P_j)=\mu_P(1-\mu_P)/(a+b+1)\). The extra term is zero for \(n=1\), and positive for \(n>1\) when \(P_j\) varies. The same shared risk also explains the within-trial correlation: for distinct patients \(i\) and \(k\), conditional independence gives \(E(Y_{ij}Y_{kj}\mid P_j)=P_j^2\), so

Independent Patient Risks Are Not the Same Hierarchy

A common modeling confusion is to equate any patient-to-patient risk variation with beta-binomial overdispersion. Consider instead risks \(P_i\) drawn independently from a beta distribution, with \(Y_i\mid P_i\sim\operatorname{Bernoulli}(P_i)\), independently for each patient. After integrating out each separate \(P_i\), the outcomes remain independent Bernoulli variables with probability \(E(P_i)\). Their total is binomial, not beta-binomial. The mixture changes each patient's risk distribution but does not create the shared within-trial dependence that inflates the count variance.

In practice, measured patient characteristics can also produce different risks. A model that conditions on those characteristics may have different probabilities \(p_i\) but still conditionally independent outcomes; its count is generally Poisson-binomial rather than binomial. A fair overdispersion check should compare the replicated outcome distribution with the model's appropriate conditional benchmark, not automatically with a binomial distribution that assumes identical risks.

Conducting a Prior Predictive Check

A useful check begins by stating the replication unit and the clinical feature to be assessed. If the prior describes variation in underlying risk across trials, generate a fresh risk for each replicated trial, then generate its patient outcomes. If it describes a single common unknown risk for one planned trial, simulate that trial accordingly; do not silently redraw the risk for every patient. Replicated outcomes should follow the model's full hierarchy.

Summaries may include the mean and variance of replicated counts, the range of rates, or the proportion of replicates exceeding a clinically specified count. Compare like with like: a count variance has units of events squared, whereas a rate variance is dimensionless. If all replicates have equal \(n\), count summaries are straightforward; if sizes differ, compare rates or account for the different binomial variances.

Diagnostics, Alternatives, and Limitations

Warning signs include predictive counts repeatedly clustering near zero and near \(n\), a large gap between the average count and the range most trials occupy, or frequent high event burdens that conflict with clinical knowledge. A large prior-predictive count variance may be appropriate when genuine trial-level heterogeneity is expected; it is not automatically a defect. Conversely, a count variance near the binomial benchmark does not prove that the hierarchy is correct: different distributions can have the same variance but different tails or shapes.

The fixed-probability binomial is a simple alternative when patients are exchangeable with a common risk and conditional independence is reasonable. A beta-binomial model allows a shared trial risk to vary across replicated trials. When patient-level risks are explained by observed covariates, a model with patient-specific probabilities may be more suitable; its count benchmark should reflect those probabilities. These approaches answer different assumptions about the source of variation, rather than providing interchangeable adjustments.

A prior predictive check is limited by the model and prior being checked. It cannot identify a unique cause of overdispersion, and summaries of counts discard patient-level structure. It also does not establish that future outcomes will follow the model. Use clinical knowledge to set plausible ranges and replication units, and keep separate the question of whether prior-generated outcomes are plausible from the question of how well a model fits observed data.

Common Pitfalls

• Claiming extra count variance when \(n=1\). A single beta-mixed Bernoulli outcome has probability \(E(P)\) and is exactly the same marginal distribution as a Bernoulli outcome with that fixed probability.
• Drawing a new risk for every patient but interpreting it as a shared trial effect. Independent patient risks yield independent marginal outcomes and a binomial count when their means are equal. A shared draw is what induces the beta-binomial dependence.
• Calling all risk heterogeneity overdispersion. Heterogeneous but independent risks do not necessarily inflate counts relative to the correct independent-outcome benchmark.
• Treating a simulated extreme as proof of failure. A few unusual draws can occur by chance. Assess the model's broader predictive behavior and whether its plausible extremes are clinically defensible.