Worked Example: Dropping Arms in a Binary Platform

Interim Decisions in a Binary Platform

A binary-response platform may compare several experimental treatments with a common control. At an interim analysis, the relevant question for an arm-dropping rule can be whether the treatment’s response probability is sufficiently likely to be below the control response probability. This is a posterior comparison, not a test of whether either observed response rate is smaller. The posterior uncertainty in both rates matters.

This tutorial works through a prespecified rule: drop an experimental arm if its posterior probability of inferiority to control is at least a chosen cutoff. The calculations use a common binary endpoint, with response coded as 1 and nonresponse as 0. All data below are hypothetical. The decision is conditional on the model, the observed interim data, and the specified cutoff; it is not by itself a guarantee of a particular long-run error rate or clinical benefit.

Definition: Let \(p_0\) be the response probability under control and \(p_j\) the response probability for experimental arm \(j\). The posterior inferiority probability for arm \(j\) is \(q_j=P(p_j<p_0\mid\text{interim data})\). Under a rule with cutoff \(c\), drop arm \(j\) if \(q_j\ge c\); otherwise retain it for the next stage, subject to the platform’s other prespecified constraints.

Model and Assumptions

For each arm \(i\), suppose the enrolled participants’ responses are conditionally independent Bernoulli observations with a common arm-specific probability \(p_i\). If \(y_i\) of \(n_i\) participants have responded by the interim, the likelihood is proportional to \(p_i^{y_i}(1-p_i)^{n_i-y_i}\). A beta prior, \(p_i\sim\operatorname{Beta}(a_i,b_i)\), is conjugate, giving the posterior

p_i\mid\text{data}\sim\operatorname{Beta}(a_i+y_i,\ b_i+n_i-y_i).

The calculations below assume that the response definition and assessment window are the same across arms; the recorded interim outcomes are valid and available for the analysis; and, conditional on each arm’s response probability, participants’ responses follow the stated Bernoulli model. They also use independent priors for the arm-specific probabilities and separate arm-specific likelihoods. Under those assumptions, the posterior distributions of the arm probabilities are independent. If a design uses a hierarchical or otherwise dependent prior, the same comparison should instead be computed from that joint posterior.

A shared control does not mean that the control data are counted once for every comparison. The control posterior is estimated from the control observations and is used in each treatment-versus-control comparison. Because the same uncertain \(p_0\) appears in all comparisons, the resulting inferiority probabilities are generally dependent across arms.

Worked Example: Two Experimental Arms and One Shared Control

At a hypothetical interim, 28 participants have assessable outcomes in each of three arms. The control has 16 responses, treatment A has 8, and treatment B has 12. Use independent \(\operatorname{Beta}(1,1)\) priors for all three response probabilities. The platform’s specified rule drops an experimental arm when its posterior inferiority probability is at least \(0.95\), while retaining the control.

Update the three posteriors. With a uniform beta prior, the posterior parameters are one plus the response count and one plus the nonresponse count:

p_0\mid\text{data}\sim\operatorname{Beta}(17,13),\qquad p_A\mid\text{data}\sim\operatorname{Beta}(9,21),\qquad p_B\mid\text{data}\sim\operatorname{Beta}(13,17).

For example, control has \(16\) responses and \(28-16=12\) nonresponses, so its posterior is \(\operatorname{Beta}(1+16,1+12)=\operatorname{Beta}(17,13)\). The corresponding posterior means are \(17/30=0.5667\) for control, \(9/30=0.3000\) for A, and \(13/30=0.4333\) for B. These means describe each arm separately; the decision rule requires the posterior probabilities of pairwise inferiority.

Calculate A’s inferiority probability. The comparison is \(P(p_A<p_0\mid\text{data})\). For these integer beta parameters, the beta-binomial identity derived below gives a tail probability with \(n=9+21-1=29\) and mixing distribution \(\operatorname{Beta}(17,13)\):

P(p_A<p_0\mid\text{data}) =P(K\ge 9),\quad K\sim\operatorname{BetaBinomial}(29,17,13) \approx 0.982.

Thus \(q_A\approx0.982\), which exceeds \(0.95\); the rule drops A.

Calculate B’s inferiority probability. Here the beta CDF threshold is \(13\), since B’s posterior is \(\operatorname{Beta}(13,17)\). Therefore

P(p_B<p_0\mid\text{data}) =P(K\ge 13),\quad K\sim\operatorname{BetaBinomial}(29,17,13) \approx 0.857.

Hence \(q_B\approx0.857\), which is below \(0.95\); B is retained under this rule. The interim action is therefore to drop A and retain B and control. This is an application of the stated armwise criterion, not a conclusion that B is superior or that its response probability is clinically adequate.

Why the Beta-Binomial Tail Gives the Comparison

The direct calculation conditions on the control probability. If the treatment posterior is \(\operatorname{Beta}(a_j,b_j)\) and the control posterior is \(\operatorname{Beta}(a_0,b_0)\), with independent posterior probabilities, then

P(p_j<p_0\mid\text{data}) =\int_0^1 F_{\operatorname{Beta}(a_j,b_j)}(u) f_{\operatorname{Beta}(a_0,b_0)}(u)\,du.

Here \(F\) is a cumulative distribution function and \(f\) is a density. The integral averages the treatment’s probability of being below a fixed value \(u\) over the uncertainty in the control probability. It is not enough to compare posterior means: that comparison omits uncertainty in both probabilities.

For positive integer \(a_j,b_j\), the beta CDF has a binomial-tail representation. Set \(m=a_j+b_j-1\). For \(0\le u\le1\),

F_{\operatorname{Beta}(a_j,b_j)}(u) =\sum_{k=a_j}^{m}{m\choose k}u^k(1-u)^{m-k} =P\{\operatorname{Binomial}(m,u)\ge a_j\}.

To see how this yields the identity, let \(U\) have the control posterior distribution and, conditionally on \(U\), let \(K\mid U\sim\operatorname{Binomial}(m,U)\). Averaging the binomial tail over \(U\) gives exactly the comparison integral:

P(p_j<p_0\mid\text{data}) =E\!\left[P(K\ge a_j\mid U)\right] =P(K\ge a_j), \quad K\sim\operatorname{BetaBinomial}(m,a_0,b_0).

The beta-binomial distribution here is the binomial distribution mixed over \(U\sim\operatorname{Beta}(a_0,b_0)\). Its probability mass at \(k\) is \({m\choose k}B(k+a_0,m-k+b_0)/B(a_0,b_0)\). This identity provides a convenient exact evaluation for integer beta parameters; numerical integration of the original one-dimensional integral is also valid. In either case, the direction of the tail matters: the threshold is \(a_j\), the treatment posterior’s first beta parameter.

Worked Example: A Comparison Just Above the Cutoff

In a separate hypothetical binary platform, an interim analysis gives control posterior \(\operatorname{Beta}(19,13)\) and experimental-arm posterior \(\operatorname{Beta}(11,21)\). Suppose the prespecified dropping cutoff is \(0.95\). These posteriors could result from independent uniform priors and 18 responses among 30 control participants, and 10 responses among 30 treatment participants.

The posterior means are \(19/(19+13)=0.5938\) for control and \(11/(11+21)=0.3438\) for treatment. To apply the rule, use the full posterior comparison. The treatment beta CDF has \(m=11+21-1=31\) and threshold \(11\), while the control posterior supplies the beta-binomial mixing parameters:

P(p_j<p_0\mid\text{data}) =P(K\ge11),\quad K\sim\operatorname{BetaBinomial}(31,19,13) \approx0.978.

Since \(0.978\ge0.95\), this rule drops the arm. The posterior probability is not obtained by subtracting the two posterior means, nor is \(0.978\) the probability that the null hypothesis is true. It is the posterior probability of the stated ordering, conditional on the model and data.

Worked Example: A Stronger Inferiority Probability

Consider another hypothetical interim with control posterior \(\operatorname{Beta}(25,17)\) and treatment posterior \(\operatorname{Beta}(14,28)\). With independent uniform priors, these correspond, for example, to 24 responses among 40 control participants and 13 responses among 40 treatment participants. Suppose the rule’s cutoff is \(0.90\).

The treatment CDF has \(m=14+28-1=41\) and tail threshold \(14\). Mixing the binomial probability over the control posterior gives

P(p_j<p_0\mid\text{data}) =P(K\ge14),\quad K\sim\operatorname{BetaBinomial}(41,25,17) \approx0.992.

Because \(0.992\ge0.90\), the arm meets the dropping criterion. This example illustrates that the threshold is a design choice specified before applying the rule: the calculated probability is evidence from the posterior, while the cutoff determines the operational action.

Applying the Rule Across Arms

At an interim with several experimental arms, calculate one posterior inferiority probability per arm using the same control posterior, then compare each value with the prespecified cutoff. The process is armwise even when more than one arm can be removed at the same interim.

Confirm the analysis set.
Check that the binary endpoint, outcome window, and interim data cut are defined consistently across arms, and that the participants included have assessable outcomes under the rule.

Update each response probability.
Combine each arm’s response and nonresponse counts with its specified prior to obtain the posterior distribution.

Evaluate each comparison.
Compute \(P(p_j<p_0\mid\text{data})\) for every experimental arm, integrating over both posterior distributions or using the beta-binomial identity when applicable.

Apply the prespecified cutoff and constraints.
Drop arms meeting the cutoff and retain those that do not, while enforcing any separate control-retention or operational constraints specified by the platform.

The comparisons share the control posterior, so their probabilities are not generally independent. For example, unusually low posterior values of \(p_0\) make it harder for every treatment arm to satisfy \(p_j<p_0\), while unusually high values make it easier. It is therefore incorrect to multiply armwise inferiority probabilities to obtain a probability that all arms are inferior. A rule that evaluates each arm against a cutoff is also not automatically a familywise frequentist error-control procedure; operating characteristics of the complete design need separate evaluation.

Diagnostics, Alternatives, and Limitations

Before acting on a calculation, verify the posterior parameters and the comparison direction. With a \(\operatorname{Beta}(a,b)\) prior, the updated parameters are \(a+y\) and \(b+n-y\), not \(a+n-y\) and \(b+y\). The event for inferiority is \(p_j<p_0\); reversing the inequality changes the decision probability. When using the beta-binomial identity, check that \(m=a_j+b_j-1\), that the tail begins at \(a_j\), and that the mixing parameters are the control posterior parameters.

Also check that the analysis uses the intended data cut and outcome definition. If response ascertainment is incomplete or differs by arm, a simple analysis of observed binary outcomes may not represent the planned model. If the prior is informative, report it and assess whether the decision is sensitive to reasonable prior choices. Small interim samples can leave posterior probabilities strongly affected by the prior, even when observed response proportions look separated.

This posterior comparison differs from comparing posterior means, which gives only a summary of each arm separately. It also differs from a posterior superiority rule, whose event has the opposite ordering, and from a predictive-probability rule, which concerns future data or a future decision. The present rule answers only the specified posterior ordering question. It does not incorporate a clinical margin unless the rule is defined to do so, and it does not establish efficacy, safety, or a causal effect without the design and model assumptions supporting those interpretations.

Common Mistake

Correction: A common error is to say, “Treatment A’s posterior mean is below control’s, so A must be dropped,” or to interpret \(q_A\) as the probability that the null is true. A lower posterior mean does not determine the probability of the ordering; compare the full posterior distributions. The correct interpretation is, for example, “Given the model and interim data, the posterior probability that \(p_A<p_0\) is approximately 0.982.” Apply the dropping rule only by comparing that probability with its prespecified cutoff.

A related error is to treat a cutoff such as \(0.95\) as a 5% frequentist type I error rate. It is a threshold on a posterior probability. The probability of making particular wrong decisions over repeated trials depends on the full design, including the prior, accrual, interim timing, endpoint model, and decision rules.

Check Your Understanding

Answer each question, then compare your reasoning with the short solutions.

With a \(\operatorname{Beta}(1,1)\) prior, an arm has 7 responses among 20 participants. What is its posterior distribution?
For treatment posterior \(\operatorname{Beta}(9,21)\) and control posterior \(\operatorname{Beta}(17,13)\), what beta-binomial tail represents \(P(p_j<p_0\mid\text{data})\)?
If an arm’s posterior inferiority probability is \(0.857\), is it dropped under a cutoff of \(0.95\)? What does the value mean?
Why is it not valid to multiply two treatment-versus-control inferiority probabilities to obtain the probability that both treatments are inferior?

Solutions. (1) There are 13 nonresponses, so the posterior is \(\operatorname{Beta}(1+7,1+13)=\operatorname{Beta}(8,14)\). (2) Here \(m=9+21-1=29\), and the control mixing distribution is \(\operatorname{Beta}(17,13)\), so the probability is \(P(K\ge9)\) for \(K\sim\operatorname{BetaBinomial}(29,17,13)\). (3) No: \(0.857<0.95\), so the arm is retained under that rule. The value is the posterior probability of \(p_j<p_0\), conditional on the model and data. (4) Both comparisons use the same uncertain control probability \(p_0\), so the events are generally dependent; their joint probability is not generally the product of their marginal probabilities.