Common Mistake: Dropping an Arm After a Lucky Start

Why an Early Response Rate Can Mislead

In a multiarm trial, an arm can look poor after only a few patients have had their outcomes observed. A low response rate may reflect a genuinely inadequate treatment, but it may also be an unlucky sequence of outcomes. The danger is especially clear when arm removal is irreversible: once the arm closes, later data cannot reveal whether its early record was a misleading fluctuation. A different kind of luck can contribute too. If a competing arm has an unusually strong early response rate, the first arm may look weak by comparison even when its own response rate remains clinically promising.

The appropriate response is not to prohibit early decisions. It is to define what evidence would justify dropping an arm, account for the uncertainty in a small sample, and specify the rule before looking at interim outcomes. This tutorial focuses on a simple Bayesian rule for a binary response endpoint. The rule is an illustration of how to make uncertainty explicit, not a universal stopping boundary.

Definition: Let \(p_j\) be the probability of response for patients assigned to arm \(j\), under the trial’s stated response definition and observation window. An arm-dropping rule specifies, in advance, the interim times and evidence required to remove arm \(j\) from further recruitment. A posterior-probability rule might drop the arm when \(P(p_j < p_{\min}\mid\text{observed data})\) exceeds a prespecified cutoff \(\gamma\), where \(p_{\min}\) is a clinically unacceptable response rate and \(\gamma\) is a high probability threshold.

This example assumes that, conditional on \(p_j\), each patient’s binary response in arm \(j\) is an independent Bernoulli observation with the same response probability. Let \(n_j\) be the number of patients with observed outcomes and \(y_j\) the number who responded. A beta prior, \(p_j\sim\operatorname{Beta}(a_0,b_0)\), represents uncertainty about the arm’s response rate before these outcomes are used. The beta distribution is supported on rates between zero and one. Its parameters must be chosen with clinical and design knowledge in mind; a convenient prior is not automatically a well-calibrated one.

The simple model also assumes the response definition and assessment window are consistent, and that the included outcomes are sufficiently mature to classify. If outcomes are pending, treating them as nonresponses would distort the observed rate. If response probabilities change across calendar time or patient subgroups, a single common \(p_j\) may not be an adequate description. Those are design and model issues to examine, not problems that a posterior cutoff repairs by itself.

From the Data to Posterior Evidence

For one arm, write \(n=n_j\), \(y=y_j\), and \(p=p_j\). Under the Bernoulli model, the likelihood for \(p\), up to a factor that does not depend on \(p\), is proportional to \(p^y(1-p)^{n-y}\). Multiplying this likelihood by the beta prior gives the posterior distribution.

\begin{aligned} \pi(p\mid y,n) &\propto p^y(1-p)^{n-y}\,p^{a_0-1}(1-p)^{b_0-1}\\ &=p^{a_0+y-1}(1-p)^{b_0+n-y-1},\\ p\mid y,n&\sim\operatorname{Beta}(a_0+y,\ b_0+n-y). \end{aligned}

The posterior tail probability \(P(p<p_{\min}\mid y,n)\) is the area under this posterior distribution below the clinically unacceptable rate. It is not the observed response rate, and it is not the probability that the treatment is inferior to another arm. A decision rule turns this evidence into an action; the posterior probability alone does not dictate which action is appropriate.

Rule template: At each prespecified interim look, drop arm \(j\) for inadequate response only if \(P(p_j<p_{\min}\mid\text{observed data})>\gamma\). Specify the prior, \(p_{\min}\), \(\gamma\), the look schedule, the response window, and the action when the rule is or is not met. The choice of these quantities is a clinical and design decision, not a result supplied by the beta-binomial model.

Worked Examples

Worked Example: One Response in Five Patients

Suppose a hypothetical arm has 1 response among 5 patients with mature outcomes. Before the trial, use \(p\sim\operatorname{Beta}(1,1)\), a uniform prior. For illustration, define rates below \(p_{\min}=0.20\) as clinically unacceptable and use \(\gamma=0.95\) as the dropping cutoff.

The posterior is \(\operatorname{Beta}(1+1,1+5-1)=\operatorname{Beta}(2,5)\). For this integer-parameter beta distribution, the posterior probability below 0.20 is \(1-(1-0.20)^5(1+5(0.20))\). Substituting and calculating gives \(1-(0.80)^5(2)=1-(0.32768)(2)=0.34464\).

The observed rate is \(1/5=0.20\), but the posterior probability that the true rate is below 0.20 is about 0.3446, well below the prespecified cutoff of 0.95. Under this illustrative rule, the arm is not dropped at this look. That conclusion does not establish that the arm is effective; it says only that the stated evidence threshold for dropping it has not been met.

Worked Example: A Competing Arm Has a Lucky Start

Consider a hypothetical randomized platform with two arms assessed using the same uniform \(\operatorname{Beta}(1,1)\) prior. Arm A has 4 responses among 4 patients, while Arm B has 1 response among 5 patients. Use the same \(p_{\min}=0.20\) and \(\gamma=0.95\) as in the first example.

For Arm A, the posterior is \(\operatorname{Beta}(5,1)\). Its posterior probability of a rate above 0.40 is \(1-(0.40)^5=1-0.01024=0.98976\). This is striking evidence that Arm A’s rate exceeds 0.40 under the stated model, but it comes from only four outcomes. For Arm B, the posterior is \(\operatorname{Beta}(2,5)\), and the calculation in the first example gives \(P(p_B<0.20\mid\text{data})=0.34464\).

Arm A’s unusually strong early record may make Arm B look unattractive in a simple ranking. But a high posterior probability that Arm A exceeds 0.40 is not the same as a high posterior probability that Arm B is below 0.20. Under the specified dropping rule, Arm B does not meet the dropping criterion. A direct comparative claim would require a posterior probability about the difference between arms, and would still need a prespecified decision rule. This example shows why one arm’s lucky start should not, by itself, determine another arm’s fate.

Why Prespecification and Interim Timing Matter

A probability cutoff only has a clear decision meaning when the rest of the rule is settled before the interim data are examined. The rule should state which outcomes count, when outcomes are considered mature, how many patients trigger a look, and which prior and clinical threshold are used. It should also specify whether crossing the boundary removes the arm immediately or leads to review under a separately defined process. Changing these choices after seeing an arm’s response rate makes the decision harder to interpret and can turn an apparently objective rule into an outcome-driven judgment.

The following calculation illustrates how a fixed threshold can make a decision reproducible. With the uniform prior, if there are no responses in \(n\) patients, the posterior is \(\operatorname{Beta}(1,n+1)\). Its cumulative probability below \(c\) is \(1-(1-c)^{n+1}\), because its density is \((n+1)(1-p)^n\) and integrating from zero to \(c\) gives that expression.

P(p<c\mid y=0,n)=\int_0^c (n+1)(1-p)^n\,dp =1-(1-c)^{n+1}.

Worked Example: Applying a Locked Look at Thirteen Outcomes

Suppose a hypothetical protocol prespecifies a single look after 13 mature outcomes, a uniform \(\operatorname{Beta}(1,1)\) prior, \(p_{\min}=0.20\), and \(\gamma=0.95\). Assume that this arm has no responses among the 13 patients.

The posterior is \(\operatorname{Beta}(1,14)\). Using the derived expression, \(P(p<0.20\mid y=0,n=13)=1-(1-0.20)^{14}=1-(0.80)^{14}=1-0.0439805=0.95602\). Since \(0.95602>0.95\), the arm meets this illustrative dropping rule at the prespecified look.

For comparison, if a look had instead been specified after 12 outcomes and there were no responses, the corresponding probability would be \(1-(0.80)^{13}=1-0.0549756=0.94502\), below 0.95. The contrast does not establish that 13 is the correct sample size. It shows that the look schedule is part of the rule: the same prior, clinical threshold, and cutoff can yield a different action at a different interim time.

Define the clinical target.
Specify the response endpoint and the response rate \(p_{\min}\) below which the arm is not worth continuing.

Set the evidence rule.
Choose the prior and \(\gamma\), and state the posterior probability that triggers arm removal.

Fix the data and timing.
Specify interim looks and outcome maturity so that each decision uses the intended observations.

Check the design before use.
Evaluate how often the rule drops arms under plausible response scenarios, including arms near the clinical threshold and arms that are truly promising.

Diagnostics, Alternatives, and Limitations

Before acting on a calculated posterior probability, check that it corresponds to the intended data. Confirm the response numerator and denominator, verify that outcomes had the required follow-up, and inspect whether patients were assessed consistently. Review whether the prior is defensible for this population and endpoint, and whether conclusions change materially under other plausible priors. If early patients differ systematically from later patients, or response rates vary over calendar time, the common-rate model may hide important structure. These checks are especially important when the result lies close to \(\gamma\).

A raw-rate rule, such as dropping an arm whenever the observed rate is below 20%, is simpler but ignores how much information the rate represents. A rate of 20% from 1 response in 5 patients and a rate of 20% from 20 responses in 100 patients are equal as point estimates, but do not carry equal precision. The posterior rule makes the uncertainty explicit through the sample size and prior. A fixed, nonadaptive design avoids outcome-driven removal during recruitment, but does not offer the same opportunity to discontinue an arm early. Neither alternative is automatically preferable; the choice depends on the scientific and operational aims of the trial.

A prespecified Bayesian rule is not immune to error. A promising arm can still cross a dropping boundary after a sufficiently unlucky sequence, and an ineffective arm can remain open after a favorable sequence. Repeated interim looks, multiple arms, prior choices, response delays, and the rule’s irreversibility all affect the design’s behavior. The cutoff \(\gamma\) is not a frequentist significance level, and a posterior probability above 0.95 does not mean there is a 95% long-run chance that this particular decision is correct.

For these reasons, rule specification should be accompanied by design evaluation under scenarios that span plausible response rates, including rates near \(p_{\min}\). The aim is to understand the chance of dropping a truly promising arm, the chance of retaining an unpromising arm, and the resulting recruitment consequences. This does not require changing a rule after observing trial outcomes: the evaluation informs the choice of rule before the trial begins.

Common Mistake

Common mistake: “The competing arm had 4 responses in 4 patients, while this arm had only 1 in 5, so the weaker arm should be dropped now.” This reasoning treats two small observed rates as decisive evidence and confuses a relative ranking with evidence that the lower arm falls below a clinical threshold. The correction is to apply the prespecified rule to the relevant posterior probability, using mature outcomes and the stated model. In the worked comparison, the apparently weaker arm’s posterior probability of a rate below 0.20 was only 0.3446, not the required 0.95. A different rule could produce a different action, but its thresholds and timing should not be improvised after seeing the data.

Key takeaway: An early extreme response rate is a noisy observation, not a decision rule. Prespecify a clinically meaningful threshold, an uncertainty-aware criterion, and the timing and data requirements for each look; then evaluate the rule’s behavior before the trial uses it.

Check Your Understanding

Answer each question using the assumptions and rule definitions in this tutorial. Solutions are included below.

With a uniform prior and 0 responses in 4 patients, what is the posterior distribution, and what is the posterior probability that \(p<0.20\)?
Why does a high posterior probability that a competing arm exceeds 0.40 not establish that another arm is below 0.20?
For a prespecified rule requiring \(P(p<0.20\mid\text{data})>0.95\), what action follows from 0 responses in 13 patients under a uniform prior?
Name two data or model checks to make before applying a response-based dropping rule.
Is \(\gamma=0.95\) the significance level \(\alpha\) of a frequentist test? Explain.

Solutions. (1) The posterior is \(\operatorname{Beta}(1,5)\), and \(P(p<0.20)=1-(0.80)^5=1-0.32768=0.67232\). (2) They are different posterior statements: evidence that one arm’s rate is high does not imply that another arm’s rate is below a clinical threshold. (3) The posterior probability is \(1-(0.80)^{14}=0.95602\), which exceeds 0.95, so the arm meets the stated dropping rule. (4) For example, verify that response outcomes are mature and that the numerator and denominator are correct; also assess whether a common response probability is plausible and whether the prior is defensible. (5) No. \(\gamma\) is a cutoff for a posterior probability under a specified Bayesian model; it is not the long-run Type I error rate \(\alpha\) of a frequentist test.