Probability of Retaining a Clinically Useful Arm

Why the Target Is More Than “Retain an Arm”

An interim rule can retain an experimental arm and still fail to retain one that would be clinically worthwhile. Conversely, it may retain several arms even though only one or none meets the clinical target. A useful design question is therefore not simply whether an arm survives, but whether at least one arm whose true effect meets a prespecified clinical benchmark survives.

This probability is an operating characteristic of the selection rule under a specified pattern of true effects. It depends on the benchmark, the true effects, the interim information, the variability and dependence of the interim estimates, and the rule used to retain arms. It is not a probability that a particular arm is clinically useful after observing its interim data.

Definition: Let $\theta_i$ be the true effect for experimental arm $i$, expressed on a scale and in a direction for which larger values indicate greater benefit. Let $\delta_{\mathrm{CU}}$ be a prespecified clinically useful effect threshold. Arm $i$ is clinically useful in a specified scenario if $\theta_i \geq \delta_{\mathrm{CU}}$. Write $U=\{i:\theta_i\geq\delta_{\mathrm{CU}}\}$ for the set of clinically useful arms. If $R$ is the set retained by the interim rule, the probability of retaining at least one clinically useful arm is $P(R\cap U\ne\varnothing)$.

The threshold must be defined before evaluating the rule, with its clinical meaning and effect scale stated. For example, if positive effects represent a reduction in a harmful outcome, a threshold of $0.20$ could mean a true reduction of at least $0.20$ units on the chosen scale. The example’s units and interpretation depend on that scale. If lower values represent benefit, the effect can instead be oriented so that larger values consistently represent greater benefit.

The set $U$ is determined by the true effects in a design scenario, not by the noisy interim estimates. The benchmark and this scenario together define what “clinically useful” means for the calculation. If no arm in the scenario meets the benchmark, then the event of retaining at least one clinically useful arm is empty and its probability is zero.

Expressing the Probability for a Selection Rule

Let $\hat{\theta}_i$ denote the interim estimate of the effect for arm $i$. For each arm, define $T_i$ as the event that the selection rule retains arm $i$. Then retaining at least one clinically useful arm is the union of the retention events for arms in $U$:

P(R\cap U\ne\varnothing) = P\left(\bigcup_{i\in U}T_i\right) = 1-P\left(\bigcap_{i\in U}T_i^{c}\right).

This complement form is useful because failure to retain any clinically useful arm means that every useful arm is dropped. Its calculation depends on the joint distribution of the interim statistics and on the exact selection rule. When arms share a control group, their estimates may be dependent; in that case, multiplying separate probabilities of dropping each arm is generally not valid.

A simple special case helps show the calculation. Suppose the rule retains arm $i$ whenever $\hat{\theta}_i>c_i$, and suppose the interim estimates of the clinically useful arms are independent and normally distributed, with means $\theta_i$ and standard errors $SE_i$. The assumptions are that the effect pattern is fixed, the normal model is suitable at the interim look, the standard errors are appropriate for that look, and the rule uses the stated thresholds. Under these assumptions:

P(T_i)=\Phi\left(\frac{\theta_i-c_i}{SE_i}\right), \qquad P(R\cap U\ne\varnothing) = 1-\prod_{i\in U} \Phi\left(\frac{c_i-\theta_i}{SE_i}\right).

The product appears because, under independence, the events that each useful arm fails its threshold are independent. If independence does not hold, the union formula remains valid, but the joint probability of dropping all useful arms must be calculated using their dependence as well.

Worked Example: One Useful Arm Under a Threshold Rule

Suppose a trial has three experimental arms. On a scale where larger values represent greater benefit, the true effects are $0.32$, $0.18$, and $0.07$ units, and the prespecified clinical-usefulness threshold is $\delta_{\mathrm{CU}}=0.25$ units. At the interim look, each arm is retained if its estimated effect exceeds $c=0.12$ units. Assume the estimate for arm 1 is normally distributed with mean $0.32$ units and $SE_1=0.10$ units. For this calculation, only arm 1 is clinically useful; the distributional assumptions for its estimate are specified directly, so no independence assumption involving the other arms is needed.

For arm 1, the probability of passing the retention threshold is

P(T_1) = P(\hat{\theta}_1>0.12) = \Phi\left(\frac{0.32-0.12}{0.10}\right) = \Phi(2.00) \approx 0.9772.

Because $U=\{1\}$, retaining at least one clinically useful arm is exactly the event that arm 1 is retained. The probability is therefore about $0.9772$, or $97.72\%$, under this specified scenario and rule. The effects for arms 2 and 3 do not enter this particular calculation because neither reaches the clinical benchmark.

This is not the probability that arm 1 is useful after its interim estimate is observed. Its usefulness was defined from its true effect in the scenario; the calculated probability describes how often the rule retains it across repeated trials under the assumed model.

Several Useful Arms: Why the Union Matters

If more than one arm meets the benchmark, the success event is not that every useful arm is retained. It is enough that at least one is retained. For independent estimates and arm-specific threshold rules, the complement formula gives a direct calculation. If estimates are dependent, a joint-normal calculation or a simulation that reproduces the specified joint distribution and selection rule can be used. A simulation estimates the probability by repeatedly generating interim data, applying the rule, and recording whether any arm in $U$ survives.

Worked Example: Two Useful Arms With Independent Estimates

Consider two arms with true effects $0.24$ and $0.21$ units, both meeting a prespecified threshold of $\delta_{\mathrm{CU}}=0.20$ units. The rule retains arm $i$ if $\hat{\theta}_i>0.15$ units. Assume independent normal interim estimates with standard errors $SE_1=0.10$ and $SE_2=0.12$ units. The two arms’ estimates satisfy the assumptions for the threshold calculation, including independence.

The probabilities of dropping the arms are

P(T_1^c) = \Phi\left(\frac{0.15-0.24}{0.10}\right) = \Phi(-0.90) \approx 0.1841,

P(T_2^c) = \Phi\left(\frac{0.15-0.21}{0.12}\right) = \Phi(-0.50) \approx 0.3085.

The probability of dropping both useful arms is their product, by the stated independence assumption. Therefore,

P(R\cap U\ne\varnothing) = 1-(0.1841)(0.3085) \approx 1-0.05679 = 0.9432.

Thus, the probability of retaining at least one of these two clinically useful arms is approximately $94.32\%$. This is not the probability of retaining both: the probability of retaining both is approximately $(1-0.1841)(1-0.3085)=0.9432? $ No: multiplying the separate retention probabilities gives $(0.8159)(0.6915)\approx0.5642$. The distinction matters: “at least one” includes outcomes in which only one arm survives.

Ranking Rules and a Direct Derivation

Some rules retain a fixed number of arms based on their interim ranking rather than applying a separate threshold to each arm. The useful-arm probability can then be an ordering probability. For two arms with independent normal estimates, retaining only the arm with the larger estimate retains arm 1 precisely when $\hat{\theta}_1-\hat{\theta}_2>0$. The difference is normally distributed, so its mean and variance give the desired probability directly.

Derivation: If $\hat{\theta}_1$ and $\hat{\theta}_2$ are independent normal estimates with means $\theta_1,\theta_2$ and standard errors $SE_1,SE_2$, then $D=\hat{\theta}_1-\hat{\theta}_2$ is normal with mean $\theta_1-\theta_2$ and variance $SE_1^2+SE_2^2$. Therefore, for a rule retaining the larger estimate, the probability of retaining arm 1 is $$ P(D>0) = \Phi\left( \frac{\theta_1-\theta_2}{\sqrt{SE_1^2+SE_2^2}} \right). $$ The result follows by standardizing $D$: subtract its mean and divide by its standard deviation, giving a standard normal random variable.

Worked Example: A Clinically Useful Arm Competes for the Top Rank

Suppose arm A has true effect $0.35$ units and arm B has true effect $0.20$ units. The clinical-usefulness threshold is $0.30$ units, so A is useful and B is not. At interim, the rule retains only the arm with the larger estimated effect. Assume independent normal estimates with $SE_A=SE_B=0.10$ units. The normality, independence, and equal-look specification are assumptions of this hypothetical calculation.

The estimated difference $D=\hat{\theta}_A-\hat{\theta}_B$ has mean $0.35-0.20=0.15$ units and variance $0.10^2+0.10^2=0.0200$ square units. Arm A is retained exactly when $D>0$. Using the unrounded standard deviation:

P(\text{at least one useful arm retained}) = P(D>0) = \Phi\left(\frac{0.15}{\sqrt{0.0200}}\right) \approx 0.8556.

The probability is approximately $85.56\%$. In this two-arm scenario, A is the only useful arm and only one arm is retained, so the event of retaining at least one useful arm is exactly the event of ranking A above B. If both arms met the benchmark, retaining either one would satisfy the “at least one” criterion; the success event would then differ from retaining the best arm.

Calculating, Checking, and Interpreting the Result

Specify the clinical benchmark.
State the effect scale, benefit direction, and prespecified threshold. Use this definition to identify the useful arms in each true-effect scenario.

State the interim model.
Give the distribution of the interim estimates, their standard errors, and any dependence assumptions. Confirm that these match the look and design being evaluated.

Apply the actual retention rule.
Translate the rule into retention events. A per-arm threshold, a top-ranked rule, and a rule retaining several arms define different events.

Evaluate the union event.
Calculate the probability that at least one arm in the useful set is retained, accounting for dependence when needed.

Report the scenario and interpretation.
Give the benchmark, true-effect pattern, rule, model assumptions, and probability together. The result is an operating characteristic, not a guarantee for an individual trial.

A diagnostic check is to verify that the useful set was formed from true effects, not interim estimates, and that the calculation corresponds to the rule actually intended. For a threshold rule, check each standardized threshold, such as $(c_i-\theta_i)/SE_i$, including its sign. If a true effect is comfortably above its retention threshold, its individual retention probability should be above one half under the normal model. For a ranking rule, check that the difference is oriented so that a positive value means the useful arm wins.

The independence shortcut is a frequent source of error. With shared controls or other dependence, the individual probabilities can be correct while their product is wrong. The complement event is still “all useful arms are dropped,” but its probability must use the joint distribution. Another check is to compare the requested event with the reported one: the probability of retaining at least one useful arm is generally not the probability of retaining all useful arms, retaining any arm, or retaining the single best arm.

Common Mistakes and Scope

Defining usefulness from the interim estimate. That makes usefulness a data-dependent label and changes the event. Define it by the prespecified benchmark applied to the true effect in the scenario.
Reporting overall retention instead. A high chance of retaining some experimental arm does not ensure that a useful arm survives. Restrict the union event to arms in $U$.
Multiplying failure probabilities without checking dependence. The product formula requires independence. Shared-control estimates commonly violate that assumption.
Confusing “at least one” with “all.” If two arms are useful, retaining one is sufficient for the target probability; retaining both is a different operating characteristic.

This probability complements the probability of retaining the best arm: it asks whether the rule preserves any arm that crosses a meaningful clinical benchmark, even when that arm is not the strongest in the scenario. It also differs from an arm’s probability of passing a statistical testing boundary. The present calculation concerns interim selection and clinical usefulness as defined here; it does not by itself establish efficacy or error-rate control.

The result is conditional on the effect pattern and model assumptions used. It does not summarize all possible true effects unless a distribution over those effects has separately been specified. Normal approximations, estimated standard errors, missing outcomes, or an operational rule different from the one modeled can alter the probability. A transparent report should therefore present the benchmark, the scenario, the selection rule, and the uncertainty model alongside the numerical result.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

One useful arm has $\theta=0.28$ units and $SE=0.10$ units. It is retained when $\hat{\theta}>0.18$ units. Under a normal interim model, what is its retention probability?
Two independent useful arms have effects $0.30$ and $0.25$ units. Both are retained when their estimates exceed $0.10$ units, and both standard errors are $0.10$ units. What is the probability of retaining at least one?
A top-one rule compares two independent normal estimates. Arm A has $\theta_A=0.34$, arm B has $\theta_B=0.24$, and their standard errors are $0.08$ and $0.06$ units. The usefulness benchmark is $0.30$ units. What is the probability of retaining a useful arm?
If no arm’s true effect reaches the prespecified clinical-usefulness benchmark, what is the probability of retaining at least one useful arm, and how should it be interpreted?

Solutions. (1) The standardized threshold is $(0.18-0.28)/0.10=-1$. Thus $P(\hat{\theta}>0.18)=\Phi(1)\approx0.8413$. (2) The useful arms’ drop probabilities are $\Phi(-2)\approx0.02275$ and $\Phi(-1.5)\approx0.06681$. By independence, the probability both are dropped is $(0.02275)(0.06681)\approx0.001520$, so the probability at least one is retained is $1-0.001520\approx0.9985$. (3) A is the only useful arm, and the difference has mean $0.34-0.24=0.10$ units and standard deviation $\sqrt{0.08^2+0.06^2}=0.10$ units. The probability A ranks first is $\Phi(0.10/0.10)=\Phi(1)\approx0.8413$. (4) The probability is zero because the useful set is empty in that scenario. This is a statement about the defined event under that true-effect pattern, not a probability that the trial has failed or that an arm would never be beneficial under another scenario.