Calculating Arm-Retention Probabilities

What an Arm-Retention Probability Measures

At an interim analysis, a prespecified rule determines whether each experimental arm continues. Before observing interim data, the rule’s outcome is uncertain. The arm-retention probability quantifies that uncertainty for a particular arm under specified true treatment effects and a specified model for the interim data. It is a design or operating-characteristic calculation: it is not the probability that the arm’s treatment is effective after seeing its data.

Write \(\Delta_k\) for the true effect of experimental arm \(k\) relative to control, with positive values denoting benefit. Let \(R_k\) be the event that arm \(k\) meets the continuation rule at the interim analysis. Under a specified model indexed by the true effects, its retention probability is \(P_{\boldsymbol{\Delta}}(R_k)\). The subscript indicates that the probability is evaluated under those assumed true effects, not estimated from the interim result.

This tutorial focuses on a one-sided threshold rule for an outcome where larger treatment-control differences are better. The threshold can be expressed in effect units, such as a minimum improvement of 3 points, or in standardized-score units. More complicated rules, such as rules combining multiple gates, require calculating the probability of their full event; a single-threshold formula should not be applied unless that event really is equivalent to one threshold.

Normal Approximation for a Continuous Outcome

Suppose the interim endpoint is continuous. For arm \(k\), let \(\bar{X}_k\) and \(\bar{X}_0\) be the interim sample means in that arm and the shared control, and define \(\hat{\Delta}_k=\bar{X}_k-\bar{X}_0\). Assume independent participants, normally distributed outcomes within each group, and a common known standard deviation \(\sigma\). Let the interim sample sizes be \(n_k\) and \(n_0\). Then

The normal model is exact under these distributional assumptions. If outcomes are not normally distributed, the distribution of the sample-mean difference may still be approximated as normal when sample sizes are sufficiently large and variances are finite; the quality of that approximation should be checked for the actual design. For planning calculations, the variance inputs are specified as part of the assumed data-generating model.

If the rule continues arm \(k\) when \(\hat{\Delta}_k>d_k\), standardizing the threshold gives

where \(\Phi\) is the cumulative distribution function of a standard normal random variable. The numerator compares the threshold with the assumed true effect; the denominator expresses that gap in standard-error units. A larger true effect or smaller standard error generally raises the probability of crossing a fixed threshold.

Thresholds on a Standardized Score

Some rules specify a minimum standardized score rather than a threshold in outcome units. Define \(Z_k=\hat{\Delta}_k/SE_k\). Under the normal model above, \(Z_k\) has variance 1 and mean \(\Delta_k/SE_k\). For a rule \(Z_k>c\), where \(c\) is a prespecified cutoff,

This expression makes clear that a standardized cutoff is not the same as a fixed effect threshold: changing the sample size changes \(SE_k\), and therefore changes the standardized mean implied by the same true effect. The cutoff and the statistic used by the actual rule must match exactly.

Binary Outcomes and a Proportion-Difference Approximation

For a binary outcome, let \(\hat{p}_k\) and \(\hat{p}_0\) be the observed proportions with the outcome in arm \(k\) and control. Suppose larger proportions are beneficial, and let \(\Delta_k=p_k-p_0\), where \(p_k\) and \(p_0\) are the assumed true proportions. With independent participants and fixed group sizes, the estimated difference has mean \(\Delta_k\) and, under a normal approximation, variance

This approximation is most credible when the expected numbers of outcomes and non-outcomes in each group are not small. If the rule is \(\hat{p}_k-\hat{p}_0>d_k\), use the resulting standard error in the same threshold formula. This is a model-based approximation, not an exact binomial calculation; discreteness can matter, especially with small samples or thresholds near attainable values.

Why Marginal Probabilities Need Careful Interpretation

For each arm, the calculation above is marginal: it considers that arm’s continuation event without requiring a particular outcome for any other arm. In a multi-arm design with a shared control, arm-specific estimated differences are generally correlated because they use the same control observations. For example, with independent treatment groups and control and a common known variance, the covariance between two estimated differences is \(\sigma^2/n_0\). This dependence does not change either arm’s marginal probability under the stated model, but it matters when asking for the probability that several arms continue together or that a particular pattern of arms is retained.

Consequently, do not add individual retention probabilities to obtain the probability that at least one arm continues, or multiply them to obtain the probability that all arms continue, unless the relevant events are independent. With shared-control estimates they usually are not. This tutorial’s single-arm calculation does not determine joint retention probabilities or which arm is best.

A short derivation also explains why the formula responds sensibly to the assumed true effect. For fixed \(SE_k>0\), let \(q(\Delta_k)=1-\Phi((d_k-\Delta_k)/SE_k)\). Differentiating with respect to \(\Delta_k\) gives

where \(\phi\) is the standard normal density, which is positive for every finite argument. Thus, under this model and for a fixed threshold, the retention probability increases as the assumed true benefit increases. This is a mathematical property of this threshold rule, not a guarantee that estimated probabilities are accurate if the model inputs are wrong.

Diagnostics, Alternatives, and Limitations

Before reporting a result, check that the event in the calculation matches the operational rule, the effect direction is correct, and the threshold and standard error use compatible units. Check sample sizes and variance inputs against the planned interim analysis. For binary outcomes, inspect expected event and non-event counts; for continuous outcomes, consider whether skewness, outliers, or unequal variances could make the normal approximation unreliable. A negative estimated effect should not be silently turned into a positive one by reversing the subtraction order.

When the approximation is doubtful, alternatives include calculating the probability directly from the assumed binomial distribution for a binary endpoint, or simulating interim data from the specified outcome model and applying the exact continuation rule to each simulated data set. Simulation can accommodate rules with multiple gates, but its precision depends on the number of simulated trials and it remains conditional on the model being simulated. If variance is estimated rather than known, its uncertainty may also affect the actual retention probability; the simple formulas here do not account for that additional variation.

• Wrong reasoning: “The arm’s estimated effect exceeded the threshold in the observed trial, so its retention probability is 100%.”
• Correction: The observed decision is one realization of the interim data. A retention probability averages the rule’s pass or fail outcome over repeated data sets under specified true effects and assumptions.
• Wrong reasoning: “Each arm has a 70% retention probability, so two arms both continue with probability \(0.70^2\).”
• Correction: That multiplication requires independence. Shared-control data generally induce dependence, so a joint probability requires the joint distribution or a simulation that preserves the shared-control structure.