Overdispersion In Binary Trial Outcomes

Why Overdispersion Matters

A binomial model assumes that, within a treatment arm, each patient has the same event probability and outcomes are independent. Under those assumptions, if \(n\) patients each have event probability \(p\), the event count \(S\) has variance \(np(1-p)\). If outcomes are more alike within clinics, sites, or other groups than that model permits, the observed event counts can vary more across repeated trials than the binomial variance predicts. This is overdispersion relative to the binomial model.

That extra variability matters for operating characteristics. A Bayesian decision rule is applied to trial data, and its false-positive rate, power, and other performance measures are estimated by repeating the trial simulation. If the simulated data omit relevant clustering, the estimated frequency of decisions can be misleading. The issue is not resolved merely by using Bayesian inference: calibration depends on whether the simulation represents the variation present in the intended trial setting.

A useful model for this setting is the beta-binomial model. In a clinical trial, it can represent patients grouped in clinics: each clinic has a latent event probability, and patients in that clinic have conditionally independent outcomes given that probability. The shared clinic probability induces positive dependence among outcomes from the same clinic.

Write \(\mu=a/(a+b)\) for the mean event probability and \(\rho=1/(a+b+1)\) for the within-clinic intracluster correlation. The parameter \(\rho\) measures the correlation between two distinct outcomes in the same clinic under this model. When \(\rho=0\), the model approaches the binomial case; positive \(\rho\) increases the variability of the count. The beta distribution is a modelling assumption, not a guarantee that every form of heterogeneity or dependence in a real trial has been captured.

Deriving the Extra-Binomial Variance

The variance follows from conditioning on the clinic probability. Given \(Q_j\), the count is binomial, so its conditional mean is \(mQ_j\) and its conditional variance is \(mQ_j(1-Q_j)\). The law of total variance separates variation within clinics from variation in their latent probabilities:

To see the last equality, use \(\operatorname{Var}(Q_j)=\mu(1-\mu)\rho\) and \(E\{Q_j(1-Q_j)\}=\mu(1-\mu)-\operatorname{Var}(Q_j)\). Substitution gives \(m\mu(1-\mu)-m\operatorname{Var}(Q_j)+m^2\operatorname{Var}(Q_j)\), which is \(m\mu(1-\mu)\{1+(m-1)\rho\}\). For \(g\) independent, equally sized clinics, the total event count has variance \(g\) times this quantity. The factor \(1+(m-1)\rho\) is the variance inflation relative to an independent binomial count with the same total number of patients.

It is important to distinguish this model from independent patient-specific probabilities. If each patient independently draws a risk from the same distribution and then has an outcome conditional on that risk, the resulting patient outcomes are still independent Bernoulli variables with the common marginal event probability. That construction alone does not produce an overdispersed total count. For extra-binomial variation, the model must induce dependence, for example by sharing a latent clinic probability among patients.

Simulating Decisions Under Overdispersion

An operating characteristic is a repeated-trial probability or expectation, such as the probability that a decision rule declares treatment benefit under a specified scenario. To estimate it, simulate many complete trials from a stated data-generating model, apply the same analysis and decision rule planned for the trial, and record the result. For a false-positive rate, the scenario must specify a null condition. For example, the two arms can have the same mean event probability, while clinic probabilities vary independently between clinics within each arm.

Consider a Bayesian analysis that treats outcomes as independent within each arm, with independent \(\operatorname{Beta}(1,1)\) priors on the arm event probabilities \(p_C\) and \(p_T\). If \(x_C\) and \(x_T\) events are observed among \(n_C\) and \(n_T\) patients, respectively, the posterior distributions under that analysis are \(\operatorname{Beta}(1+x_C,1+n_C-x_C)\) and \(\operatorname{Beta}(1+x_T,1+n_T-x_T)\). Suppose the decision is to declare benefit when \(P(p_T<p_C\mid\text{data})>0.95\). This definition makes the decision rule explicit; it does not imply that its repeated-trial false-positive rate is 5%.

For each simulated replicate, generate clinic probabilities from the specified beta distribution, generate patient outcomes conditional on those probabilities, calculate the posterior probability using the planned analysis, and apply the cutoff. Under a null scenario, the fraction of replicates crossing the decision threshold estimates the false-positive rate. The simulated outcome model and the analysis model need not be the same: comparing them is precisely how one can assess the calibration of a planned rule under extra-binomial variation.

A careful calibration study can also vary the decision cutoff while keeping the scenario and data-generation procedure fixed. A more stringent posterior-probability cutoff will generally lead to fewer declarations, though the resulting rate must be measured rather than assumed. If a cutoff is selected using one batch of simulations, its performance should not be reported as if it had been independently validated by that same batch.

Model Choices, Diagnostics, and Limitations

The beta-binomial model is one way to represent clustered binary outcomes; it is not the only one. The appropriate approach depends on the trial design and on the source of extra variability.

Diagnostics for a simulation study begin with the model specification, not only with its output. Check that the simulated mean event rate matches its target, that cluster sizes and allocation match the planned design, and that the generated within-clinic dependence agrees with the chosen \(\rho\). Across many replicates, compare the empirical event-count variance with the model’s predicted variance. For \(g\) clinics of size \(m\), that prediction is \(g m\mu(1-\mu)\{1+(m-1)\rho\}\). A discrepancy can signal an implementation error or a mismatch between the intended and simulated model.

Also inspect decision rates across relevant null and benefit scenarios, rather than treating one simulated rate as a general property of the rule. A result can be unstable when there are few simulations, few clinics, or rare events. In particular, increasing the number of patients within a small number of clinics does not provide the same information about between-clinic variation as adding clinics. The beta-binomial model also imposes a particular form of heterogeneity: it does not automatically represent covariate effects, unequal clinic sizes, more complex dependence, or all forms of treatment-effect variation.

A common mistake is to assign each patient an independent random event probability and conclude that the resulting total count must be overdispersed. If those probabilities are independently drawn from the same distribution, the marginal outcomes are independent Bernoulli variables with a common mean, so their total count remains binomial. The correction is to specify the dependence that creates extra variation, such as a shared latent probability for patients in the same clinic, and then verify that the simulated structure matches the intended trial.