Posterior Superiority Rules for Dropping Arms

Why Use a Posterior Rule to Drop an Arm?

In a multiarm trial, an interim analysis may show that one experimental treatment is performing worse than a reference treatment. A posterior superiority rule turns that comparison into a prespecified decision: remove an arm from future randomization when the posterior probability that its treatment effect is inferior to the reference is sufficiently high. The rule is about the probability of an ordering, conditional on the observed data and the model; it is not a test that proves an arm is ineffective.

The direction of the outcome scale must be clear. In this tutorial, larger values of the arm-specific parameter $\theta_k$ represent better outcomes, and $\theta_0$ denotes the parameter for the reference arm. For a binary outcome, for example, $\theta_k$ could be a response probability. For a continuous outcome, it could be a mean. The reference may be a control or another designated comparator, but it must be chosen in advance for the rule being applied.

Definition: Let $D$ be the interim data and let $\gamma$ be a prespecified posterior threshold, typically close to 1. An experimental arm $k$ is dropped by the strict-inferiority rule if $$ P(\theta_k < \theta_0 \mid D) \geq \gamma. $$ The probability is calculated under the specified likelihood and prior, conditional on the data available at that interim analysis. Equality is included in the rule as written.

This rule uses inferiority on the parameter scale, with a boundary at equality. It does not include a clinically meaningful margin: the comparison is simply whether $\theta_k$ is lower than $\theta_0$. A high value of $\gamma$ demands strong posterior evidence before dropping an arm. Neither $\gamma$ nor the posterior probability is, by itself, a frequentist type I error rate.

Calculating a Posterior Inferiority Probability

The calculation requires a model that yields a posterior distribution for both the experimental arm and the reference. Under independent normal posteriors, their difference has a normal distribution, so the probability can be evaluated using a standard normal cumulative distribution function. In other models, the probability may be obtained by integration or by counting posterior draws.

For the normal calculation, suppose the posterior distributions are independent and satisfy $\theta_k \mid D \sim N(m_k,v_k)$ and $\theta_0 \mid D \sim N(m_0,v_0)$, where the second argument of $N$ is the variance. The contrast $\theta_k-\theta_0$ then has posterior mean $m_k-m_0$ and variance $v_k+v_0$. Therefore:

P(\theta_k < \theta_0 \mid D) = \Phi\left( \frac{m_0-m_k}{\sqrt{v_k+v_0}} \right).

Here $\Phi$ is the standard normal cumulative distribution function. The denominator is the posterior standard deviation of the contrast, not the sum of the two posterior standard deviations. If the posterior distributions of the arm parameters are dependent, the contrast variance must instead include their covariance: $\operatorname{Var}(\theta_k-\theta_0\mid D)=v_k+v_0-2\operatorname{Cov}(\theta_k,\theta_0\mid D)$.

The independence condition in the normal formula is appropriate, for example, under independent arm-specific priors and conditionally independent arm-specific likelihoods. It should not be assumed automatically: a model with shared parameters or a joint prior can induce posterior dependence. For any model, the probability must reflect the actual joint posterior of the two parameters.

Worked Example: Applying the Rule Across Three Arms

Consider a hypothetical interim analysis of a continuous outcome measured in points, with larger values better. There is one reference arm and three experimental arms, A, B, and C. Each arm has $n=25$ observations. The sample means are 50, 44, 48, and 54 points, respectively, for the reference and arms A, B, and C. Assume independent priors $\theta_j \sim N(50,100)$ for all four arms, and an observation model $X_{ij}\mid\theta_j\sim N(\theta_j,100)$ with known variance $100$ points squared. The arm-specific likelihoods are independent. Use $\gamma=0.95$.

These assumptions give a normal-normal update. The posterior variance is the same in every arm:

v=\left(\frac{1}{100}+\frac{25}{100}\right)^{-1} =\frac{100}{26} =3.8462\ \text{points}^2.

The posterior mean is the precision-weighted average of the prior mean and the sample mean. Since the prior precision is $1/100$ and the data precision for a sample mean is $25/100$, the posterior means are:

m_j = \frac{50/100+25\bar{x}_j/100}{1/100+25/100} = \frac{50+25\bar{x}_j}{26}.

Thus the posterior means are 50.000 for the reference, 44.231 for A, 48.077 for B, and 53.846 for C points. Because the arm posteriors are independent, each contrast with the reference has posterior standard deviation:

\sqrt{v+v} = \sqrt{2(100/26)} = 2.7735\ \text{points}.

For arm A, the standardized difference is $(50.000-44.231)/2.7735=2.080$, using unrounded posterior values in the calculation. Therefore $P(\theta_A<\theta_0\mid D)=\Phi(2.080)\approx0.981$. Since $0.981\geq0.95$, arm A meets the dropping rule.

For arm B, the standardized difference is $(50.000-48.077)/2.7735=0.6930$, and $P(\theta_B<\theta_0\mid D)=\Phi(0.6930)\approx0.756$. Arm B does not meet the rule. For arm C, the standardized difference is $(50.000-53.846)/2.7735=-1.387$, giving $P(\theta_C<\theta_0\mid D)=\Phi(-1.387)\approx0.0828$. Arm C also does not meet the rule. Under the stated rule, only arm A is dropped at this interim analysis.

The calculation assumes the specified outcome direction, known common variance, independent normal priors, and independent arm-specific data. If these assumptions are inappropriate, the computed probabilities need not represent the intended posterior evidence. In particular, uncertainty in the outcome variance would require a model that accounts for it.

Binary Outcomes and Posterior Computation

The same decision rule applies when the endpoint is binary, but the posterior distribution may not be normal. For independent binomial outcomes and independent beta priors, the arm-specific response probabilities have beta posteriors. The inferiority probability is then an integral over the joint posterior. It can be evaluated analytically in some cases or estimated from posterior draws in more general models.

Worked Example: Exact Calculation for Binary Responses

Suppose a hypothetical interim analysis records 1 response among 2 participants in the reference arm and 0 responses among 2 participants in experimental arm A. Let $p_0$ and $p_A$ denote the response probabilities, with higher values better. Assign independent $\operatorname{Beta}(1,1)$ priors to both probabilities and use independent binomial likelihoods. Apply the rule with $\gamma=0.95$.

After observing the data, the conjugate beta updates give $p_0\mid D\sim\operatorname{Beta}(2,2)$ and $p_A\mid D\sim\operatorname{Beta}(1,3)$. Under the independent priors and likelihoods, these posterior probabilities are independent. The density of $\operatorname{Beta}(1,3)$ is $3(1-x)^2$, and the cumulative distribution function of $\operatorname{Beta}(2,2)$ is $3x^2-2x^3$. Conditioning on $p_A=x$, the probability that $p_0$ exceeds $x$ is $1-(3x^2-2x^3)$. Thus:

\begin{aligned} P(p_A<p_0\mid D) &=\int_0^1 3(1-x)^2(1-3x^2+2x^3)\,dx\\ &=1-3E(p_A^2\mid D)+2E(p_A^3\mid D)\\ &=1-3(0.10)+2(0.05)\\ &=0.80. \end{aligned}

The beta moments used here are $E(p_A^2\mid D)=1(2)/(4(5))=0.10$ and $E(p_A^3\mid D)=1(2)(3)/(4(5)(6))=0.05$. The posterior probability of inferiority is 0.80, below 0.95, so arm A is not dropped. This small example also shows why observing fewer responses in an arm does not alone determine the decision: the rule incorporates uncertainty about both response probabilities.

With posterior simulation, the same probability has a simple interpretation as the fraction of joint posterior draws in which $\theta_k<\theta_0$. For example, if 9,700 of 10,000 joint draws satisfy that inequality, the estimated probability is $9{,}700/10{,}000=0.970$. This exceeds a threshold of 0.95. The estimate has Monte Carlo standard error approximately $\sqrt{0.970(1-0.970)/10{,}000}=0.00171$; if the estimate is close to the decision threshold, more draws may be needed to make the computational approximation sufficiently precise.

Justification, Diagnostics, and Interpretation

The normal formula follows directly from the distribution of a difference. If the two posterior parameters are independent normal random variables, their difference is normal, with its mean equal to the difference in means and its variance equal to the sum of their variances. The event $\theta_k<\theta_0$ is equivalent to $\theta_k-\theta_0<0$. Standardizing that contrast gives:

\begin{aligned} P(\theta_k<\theta_0\mid D) &=P\left( \frac{(\theta_k-\theta_0)-(m_k-m_0)} {\sqrt{v_k+v_0}} < \frac{0-(m_k-m_0)}{\sqrt{v_k+v_0}} \;\middle|\;D \right)\\ &=\Phi\left(\frac{m_0-m_k}{\sqrt{v_k+v_0}}\right). \end{aligned}

This derivation depends on the stated normality and independence assumptions. For a nonnormal or dependent posterior, the definition of the rule still applies, but this particular formula does not. Use the joint posterior directly, for example by integrating the joint density or computing the proportion of posterior draws for which the experimental parameter is below the reference parameter.

Before applying the rule, check that the estimand and outcome direction are consistent across arms, the reference is the intended comparator, and the interim data included in the likelihood are the data the decision plan specifies. Examine model fit and the sensitivity of the posterior probability to plausible prior choices and modeling assumptions. For posterior simulation, check that chains mix adequately and that the estimated probability is stable enough for a decision near $\gamma$. Delayed or missing outcomes can also make an interim analysis misleading if the observed participants do not yet provide a suitable basis for the model.

A posterior threshold is a Bayesian decision criterion, not a guarantee of a particular long-run error rate. Its operating characteristics depend on the data-generating conditions, model, prior, interim timing, and rule. Trial designers should evaluate the full decision procedure under relevant scenarios before use. Dropping an arm also changes which treatments can receive future participants and can affect later comparisons; a probability calculation alone does not address every operational constraint in a platform trial.

The rule differs from adaptive allocation: allocation changes how future participants are assigned among available arms, whereas dropping an arm removes it from the set of available options. It also differs from a predictive-probability rule, which concerns the probability of a future trial outcome under continued enrollment. Here the criterion is specifically the current posterior probability that an arm’s parameter is below the reference. A posterior probability of inferiority is not the probability that the null hypothesis is true, nor does it establish that treatment caused a poor outcome beyond the trial’s randomized comparison.

Common Mistakes and Limitations

Treating the cutoff as a p-value. A rule using $\gamma=0.95$ does not automatically have a 5% frequentist type I error rate. That rate depends on repeated-sampling behavior of the complete design.
Reversing the outcome direction. If lower values are better, the event defining inferiority must be reversed or the parameter redefined. Applying $P(\theta_k<\theta_0\mid D)$ without checking direction can drop a beneficial arm.
Ignoring posterior dependence. Shared model parameters can induce dependence. In that case, adding marginal variances is not a valid contrast variance calculation; the covariance must be included, or the joint posterior used directly.
Equating a high probability with clinical irrelevance. The rule asks whether one parameter is below another, not whether the difference is clinically important. A margin-based decision is a different rule.
Applying the threshold after repeatedly checking without design evaluation. Repeated interim decisions are part of the trial design. The posterior cutoff alone does not describe all consequences of repeated looks or changing enrollment.

The principal limitation is that a seemingly simple rule inherits the assumptions of the posterior model and the design that uses it. It can be transparent and easy to communicate, but it should be prespecified, checked against plausible alternatives, and interpreted as one component of a trial decision process rather than a universal measure of treatment value.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

With larger outcome values better, an arm has $P(\theta_k<\theta_0\mid D)=0.947$. Does it meet a rule with $\gamma=0.95$?
Under independent normal posteriors, suppose $m_0=10$, $m_k=8$, $v_0=1$, and $v_k=3$. Write the inferiority probability and identify the contrast standard deviation.
Why is a posterior inferiority probability of 0.95 not automatically a frequentist type I error rate of 0.05?
For binary outcomes, why is it generally not enough to compare the observed response proportions before applying the rule?

Solutions. 1. No. The rule requires a probability at least 0.95, and $0.947<0.95$. 2. The contrast standard deviation is $\sqrt{v_0+v_k}=\sqrt{4}=2$, and $P(\theta_k<\theta_0\mid D)=\Phi((10-8)/2)=\Phi(1)\approx0.8413$. 3. The posterior probability is conditional on the observed data and specified model; a frequentist error rate is a repeated-sampling probability of a decision under specified data-generating conditions. They are different quantities. 4. Observed proportions do not represent all uncertainty about the arm-specific response probabilities. The posterior probability compares their full posterior distributions, which also depend on sample size and the prior and likelihood assumptions.

Posterior Superiority Rules for Dropping Arms

What You'll Learn