Beta-Binomial Decisions for Binary Outcomes

Binary Responses and Bayesian Decisions

For a binary outcome, each participant either experiences the response of interest or does not. The response rate in an arm is therefore a probability, and the observed data are a number of responses out of a number treated. A beta-binomial model translates those counts into a posterior distribution for each arm’s response rate. Posterior probabilities calculated from those distributions can then be compared with a decision threshold.

This tutorial considers decisions based on whether a treatment response rate exceeds the control rate, or exceeds it by a specified clinically meaningful amount. The calculations require a model for the arm-specific response probabilities and a prior distribution for each probability. They do not, by themselves, determine which clinical margin or posterior threshold is appropriate.

Definition: In arm \(j\), let \(n_j\) participants have binary outcomes and let \(y_j\) be the number with the response. Conditional on the arm’s response probability \(p_j\), assume \(Y_j\sim\operatorname{Binomial}(n_j,p_j)\). A beta-binomial model assigns \(p_j\sim\operatorname{Beta}(a_j,b_j)\), where \(a_j>0\) and \(b_j>0\). The beta distribution has density proportional to \(p_j^{a_j-1}(1-p_j)^{b_j-1}\) for \(0<p_j<1\).

The binomial model assumes that, within an arm, participants’ outcomes are conditionally independent and have the same response probability. The basic model also treats the arm-specific response probabilities as distinct parameters. Unless a dependence structure is specified, we use independent beta priors across arms. These assumptions concern the probability model; they do not establish that the trial’s allocation, outcome measurement, or follow-up is free of bias.

Posterior Updating

The binomial likelihood for \(y_j\) responses among \(n_j\) participants is proportional to \(p_j^{y_j}(1-p_j)^{n_j-y_j}\). Multiplying it by the beta prior density gives a posterior kernel with powers \(a_j+y_j-1\) and \(b_j+n_j-y_j-1\). Normalizing that kernel yields the beta posterior:

p_j\mid Y_j=y_j \sim \operatorname{Beta}(a_j+y_j,\; b_j+n_j-y_j).

The prior parameters act like fractional counts in the update: \(a_j\) contributes to the response count and \(b_j\) to the nonresponse count. This is a useful way to understand the calculation, but it does not mean that prior information is literally observed trial data. With independent priors and conditionally independent arm outcomes, the arm-specific posteriors are independent given the data. A shared control still appears in every treatment-control contrast, so those contrasts are generally dependent through uncertainty about the common control rate.

Worked Example: Updating Two Arm Response Rates

Suppose a small hypothetical trial observes 1 response among 1 participant in treatment and 0 responses among 1 participant in control. Use independent \(\operatorname{Beta}(1,1)\) priors in both arms. These uniform priors are chosen here for transparent arithmetic, not as a universal default.

For treatment, \(y_T=1\) and \(n_T=1\), so the posterior parameters are \(1+1=2\) and \(1+1-1=1\). Thus \(p_T\mid D\sim\operatorname{Beta}(2,1)\). For control, \(y_C=0\) and \(n_C=1\), giving \(p_C\mid D\sim\operatorname{Beta}(1,2)\).

The likelihood assumptions are that each recorded outcome is binary and that, conditional on its arm’s response probability, the outcome follows the stated binomial model. With one observation per arm, these assumptions cannot be assessed meaningfully from the data; the example illustrates updating, not a reliable clinical conclusion.

Posterior Superiority and Risk Differences

Let \(p_T\) and \(p_C\) denote the treatment and control response probabilities. The posterior probability of response-rate superiority is \(P(p_T>p_C\mid D)\), where \(D\) denotes the observed data. If \(f_T\) is the posterior density of \(p_T\) and \(F_C\) is the posterior cumulative distribution function of \(p_C\), independence of the two posterior rates gives:

P(p_T>p_C\mid D) =\int_0^1 f_T(x)F_C(x)\,dx.

For a clinically meaningful margin \(\delta\geq 0\), the corresponding decision probability is \(P(p_T-p_C>\delta\mid D)\). Since a risk difference cannot exceed 1, and the condition requires \(p_T>\delta\), this probability can be written as:

P(p_T-p_C>\delta\mid D) =\int_\delta^1 f_T(x)F_C(x-\delta)\,dx.

These expressions integrate over uncertainty in both response rates. In general, they are evaluated by numerical integration or posterior simulation. For beta distributions with integer parameters, beta-distribution functions can also be expressed using finite sums; the integral formulation remains valid for any positive beta parameters.

Worked Example: Probability of Exceeding a Clinical Margin

Use the posteriors from the preceding example: \(p_T\mid D\sim\operatorname{Beta}(2,1)\) and \(p_C\mid D\sim\operatorname{Beta}(1,2)\). The treatment density is \(f_T(x)=2x\), and the control cumulative distribution function is \(F_C(z)=2z-z^2\) for \(0\leq z\leq1\).

First, the posterior probability that treatment’s response rate is higher is:

\begin{aligned} P(p_T>p_C\mid D) &=\int_0^1 2x(2x-x^2)\,dx\\ &=\left[\frac{4}{3}x^3-\frac{1}{2}x^4\right]_0^1\\ &=\frac{5}{6}\approx 0.8333. \end{aligned}

Now consider the hypothetical clinical margin \(\delta=0.20\), or 20 percentage points. The posterior probability of exceeding that margin is:

\begin{aligned} P(p_T-p_C>0.20\mid D) &=\int_{0.20}^1 2x\{2(x-0.20)-(x-0.20)^2\}\,dx\\ &=\left[-\frac{1}{2}x^4+1.6x^3-0.44x^2\right]_{0.20}^1\\ &=0.6656. \end{aligned}

Under this model and prior, the posterior probability of any superiority is about 0.8333, while the probability that the response-rate advantage exceeds 20 percentage points is 0.6656. Neither number is the probability that a hypothesis is true in a frequentist sense; each is a probability statement about the response rates under the specified posterior model.

Applying a Threshold Across Several Arms

A posterior decision rule can declare arm \(k\) promising when its probability of exceeding control is greater than a prespecified threshold \(q\). For example, the rule may be \(P(p_k>p_C\mid D)>q\). The threshold is a decision criterion, not a property of the beta-binomial model. Its justification depends on the decision context and the consequences of selecting an ineffective arm or overlooking a useful one.

Worked Example: A Three-Arm Decision With Shared Control

Consider a hypothetical study with one control and two treatments. Control has 1 response among 2 participants; treatment A has 5 among 5; and treatment B has 3 among 3. Assign independent \(\operatorname{Beta}(1,1)\) priors to all three response rates. Suppose the illustrative decision rule requires a posterior superiority probability greater than \(q=0.90\).

The posterior distributions are \(p_C\mid D\sim\operatorname{Beta}(2,2)\), \(p_A\mid D\sim\operatorname{Beta}(6,1)\), and \(p_B\mid D\sim\operatorname{Beta}(4,1)\). The control cumulative distribution function is \(F_C(x)=3x^2-2x^3\). For \(X\sim\operatorname{Beta}(k,1)\), its moments are \(E(X^r)=k/(k+r)\), so:

\begin{aligned} P(X>p_C\mid D) &=E\{F_C(X)\mid D\}\\ &=3E(X^2)-2E(X^3)\\ &=\frac{3k}{k+2}-\frac{2k}{k+3}. \end{aligned}

For treatment A, substitute \(k=6\): \(3(6/8)-2(6/9)=2.25-1.3333=0.9167\). For treatment B, substitute \(k=4\): \(3(4/6)-2(4/7)=2.0000-1.1429=0.8571\). Thus A exceeds the 0.90 threshold and B does not. This is a conclusion under the specified priors, model, data, and threshold; it is not a guarantee that A has a higher response rate in the population.

Both comparisons use the same uncertain control rate. The marginal probabilities 0.9167 and 0.8571 can be calculated separately as above, but the events \(p_A>p_C\) and \(p_B>p_C\) are not independent: a posterior draw with a high control rate tends to make both comparisons less favorable. Do not multiply the two probabilities to obtain a joint probability that both treatments are superior.

Diagnostics, Alternatives, and Limitations

Before interpreting a posterior decision, check that the response is defined consistently, the denominators include the intended participants, and the recorded counts satisfy \(0\leq y_j\leq n_j\). Confirm that a binomial model is defensible: substantial clustering, repeated outcomes, or differing response probabilities within an arm may require a different likelihood. A beta prior is flexible for a single probability, but the posterior still depends on its parameters, especially when arm counts are small. Review whether the prior places plausible mass over response rates relevant to the application and whether the resulting conclusion is driven by the prior.

A posterior probability for superiority is not the same as a probability of clinically important benefit. If the decision requires a margin, calculate \(P(p_T-p_C>\delta\mid D)\), not merely \(P(p_T>p_C\mid D)\). Likewise, a posterior threshold is not automatically a frequentist type I error rate or familywise error rate. The earlier material on Bayesian decision targets and selecting posterior probability thresholds addresses the rationale for decision events and cutoffs; the beta-binomial calculation supplies probabilities for binary outcomes under the stated model.

A normal approximation to a difference in observed proportions may be convenient in some settings, but it can behave poorly with sparse counts or rates near 0 or 1. The beta-binomial approach directly represents uncertainty on the probability scale and gives valid posterior distributions within \([0,1]\). It does not remove the need to state assumptions or choose priors. Nor does a simple independent-prior model borrow information across treatment arms; modeling such borrowing is a different extension.

Common mistake: equating superiority with a meaningful advantage. A large \(P(p_T>p_C\mid D)\) can coexist with a modest probability of exceeding a clinically meaningful margin. Specify and calculate the event that matches the decision.
Common mistake: treating shared-control comparisons as independent. The treatment posteriors may be independent under independent priors, but their contrasts share \(p_C\). Use a joint posterior calculation when the decision concerns several arms simultaneously.
Common mistake: reading the threshold as an error rate. A rule that selects when a posterior probability exceeds \(q\) does not, by itself, guarantee a particular long-run false-positive rate. That operating characteristic requires separate evaluation under specified data-generating scenarios.

Key takeaway: For a binary endpoint, update each beta prior by adding observed responses and nonresponses to its parameters. Then integrate over the treatment and control posteriors for the decision event of interest, and apply a threshold only after stating the model, prior, event, and clinical context.

Check Your Understanding

Use the beta-binomial model and show how the data and event determine each answer.

With a \(\operatorname{Beta}(2,3)\) prior and 4 responses among 10 participants, what is the posterior distribution?
If \(p_T\mid D\sim\operatorname{Beta}(2,1)\) and \(p_C\mid D\sim\operatorname{Beta}(1,2)\), what is \(P(p_T>p_C\mid D)\)?
Why can two posterior superiority probabilities against a shared control not generally be multiplied to obtain the probability that both treatments are superior?
What probability should be calculated if the decision requires treatment to improve the response rate by more than 10 percentage points?

Solutions. (1) There are \(10-4=6\) nonresponses, so the posterior is \(\operatorname{Beta}(2+4,3+6)=\operatorname{Beta}(6,9)\). (2) Integrating the control cumulative distribution function gives \(\int_0^1 2x(2x-x^2)\,dx=5/6\approx0.8333\). (3) The events share the uncertain control rate and are therefore generally dependent, even if treatment rates are independent in the posterior. (4) Calculate \(P(p_T-p_C>0.10\mid D)=\int_{0.10}^1 f_T(x)F_C(x-0.10)\,dx\).