Ranking Treatment Arms With a Shared Control

Why Shared-Control Rankings Need Care

At an interim analysis, the estimated treatment effects may provide a useful ordering of the arms. But when every arm is compared with the same control group, those estimates are dependent: a control mean that happens to be low makes all estimated benefits look higher, while a high control mean makes them all look lower. The observed order is still easy to calculate, but interpreting how stable that order is requires accounting for the shared control.

This tutorial focuses on ranking arms using their estimated treatment-control differences and examining the uncertainty in pairwise differences between those estimates. A ranking is descriptive: it orders the estimates in the observed data. It does not, by itself, establish which treatment has the largest true effect or prescribe which arms should continue.

The calculations below use a simple model: participants in each group are independent, outcomes in every group are normally distributed with a common known variance \(\sigma^2\), and the treatment arms and control are observed on the same outcome scale and at the same assessment time. The sample sizes are fixed. These assumptions let us calculate exact variances and normal-based intervals. In an application, the variance is usually estimated, and the analysis must use the appropriate variance estimator and reference distribution.

How the Shared Control Affects Pairwise Ranks

For two treatment arms \(i\) and \(j\), the difference between their estimated treatment effects is \(\hat{\tau}_i-\hat{\tau}_j=(\bar X_i-\bar X_0)-(\bar X_j-\bar X_0)=\bar X_i-\bar X_j\). The common control mean cancels. Under the stated model, the estimated treatment effects have a shared-control covariance, but the uncertainty in their pairwise difference can be found directly from the two treatment-arm means.

Here \(n_i\) and \(n_0\) are the sample sizes in treatment arm \(i\) and control. The covariance is positive because both estimates contain \(-\bar X_0\). The resulting correlation is the covariance divided by the product of the two standard errors. It is not generally the same for every pair if treatment sample sizes differ.

The final equality follows by substituting the variances and covariance: the two terms involving \(1/n_0\) cancel. Thus, under this model, uncertainty in the relative effect of two arms depends on their treatment-group sample sizes and outcome variance, not on the control sample size. The control sample size still affects the marginal precision of each treatment-versus-control estimate.

With known \(\sigma\), a pointwise 95% interval for the difference in true effects, \(\tau_i-\tau_j\), is the observed difference \(\hat{\tau}_i-\hat{\tau}_j\) plus or minus \(1.96\) times its standard error. This interval describes uncertainty for that one pairwise comparison; it is not a simultaneous interval for all pairs, nor is it a probability statement about which treatment is truly best.

A Three-Arm Ranking With a Common Control

With three arms, the same principle applies to every pair. Sorting the three observed effects gives an ordering, while pairwise differences show how uncertain the adjacent and non-adjacent comparisons are. Because the estimates share a control, it would be incorrect to calculate their pairwise difference variance as if the estimates were independent.

Why the Covariance Calculation Matters

A short derivation makes clear where a common mistake arises. Write each arm’s estimate as its treatment mean minus the common control mean. Since the treatment groups and control are independent, the covariance between the two treatment means and either control mean is zero. The covariance between the estimates therefore comes from the shared control term:

Subtracting twice this covariance is essential when calculating the variance of the difference between two estimates. Omitting it incorrectly adds the shared-control variation twice instead of allowing it to cancel. Under unequal treatment sample sizes, the pairwise formula remains \(\sigma^2(1/n_i+1/n_j)\) in this equal-variance model, even though the marginal standard errors and correlations differ by arm.

Diagnostics, Alternatives, and Limitations

Before interpreting a rank, check that all estimates use the same outcome scale, contrast direction, and assessment time. If effects are reported with mixed directions—for example, a lower value is favorable for one outcome but a higher value is favorable for another—convert them to a consistent direction before sorting. Also verify the sample sizes and variance assumptions used for the uncertainty calculation. If group variances differ, the equal-variance formulas above are not appropriate without modification.

A frequent error is to treat the treatment-control estimates as independent and calculate the variance of their difference by simply adding their marginal variances. That calculation gives \(\sigma^2(1/n_i+1/n_0)+\sigma^2(1/n_j+1/n_0)\), which incorrectly retains two control-variance terms. The correct calculation subtracts twice the covariance, \(\sigma^2/n_0\), leaving \(\sigma^2(1/n_i+1/n_j)\). Another error is to say, “C ranked first, so C is the best treatment.” The observed order describes the point estimates; it is not proof of the true ordering.

Ranking point estimates is simple and remains meaningful as a description of the observed data. Pairwise differences with standard errors or intervals add information about precision. An alternative design with separate control groups for each treatment would not have this same shared-control covariance, but would use control data differently and generally require more control participants to achieve comparable precision. In either design, rankings alone do not resolve uncertainty or account for all features that may matter to an interim decision.

Pairwise intervals also have limitations. The intervals above are pointwise, not adjusted to provide simultaneous coverage for all comparisons. Overlapping intervals are not a formal test that two effects are equal, and an interval including zero does not make the observed ranking disappear. Conversely, a precise ordering in one interim data set does not guarantee that the ordering will persist with additional data. The appropriate interpretation is narrower: state the estimated order, quantify uncertainty in relevant pairwise differences, and avoid claiming more than those calculations support.