Ranking Arms With Noisy Interim Estimates

Why an Interim Ranking Is Uncertain

An interim analysis may show that one treatment arm has a larger estimated effect than another. That ordering is a feature of the observed data, not a guarantee about the arms’ true effects. If the estimates are noisy, a new sample of the same size could place the arms in the opposite order. This matters because an adaptive decision can depend on which arm appears best at the interim.

The central question here is not whether an arm has crossed a retention or efficacy threshold. It is narrower: how much confidence should be placed in the order of the estimated effects? A numerical ranking alone does not answer that question. The gap between estimates must be considered alongside their sampling variability and, when relevant, their covariance.

Definition: Let \(\theta_j\) be the true treatment effect for arm \(j\), measured on a prespecified common scale, and let \(\hat{\theta}_j\) be its interim estimate. An observed ranking orders the arms by their \(\hat{\theta}_j\) values. A true ranking orders them by their \(\theta_j\) values. Sampling variability can make these rankings differ. In this tutorial, larger effects are taken to be preferable; reverse the inequalities if smaller effects are preferable.

A ranking should use the quantity that matches the selection objective. If the objective is the largest treatment effect in outcome units, rank effect estimates—not automatically their \(z\)-statistics or \(p\)-values. Standardizing by uncertainty answers a different question: how strongly the data distinguish an effect from a reference value. With unequal standard errors, these two orderings can differ.

A Pairwise Model for Ranking Uncertainty

Consider two arms, \(A\) and \(B\), evaluated on the same endpoint at the same interim. Suppose their estimates are approximately jointly normal, centered at their true effects, with standard errors \(SE_A\) and \(SE_B\). Let \(\operatorname{Cov}(\hat{\theta}_A,\hat{\theta}_B)\) denote the covariance between the estimates. These assumptions may be reasonable for sufficiently informative samples and a suitable estimator; the covariance must reflect the actual design and analysis.

Model: The estimated difference is \(D=\hat{\theta}_A-\hat{\theta}_B\). Under the joint normal approximation, \(D\sim N(\theta_A-\theta_B,\,SE_A^2+SE_B^2-2\operatorname{Cov}(\hat{\theta}_A,\hat{\theta}_B))\). The second argument is the variance. Write \(SE_D\) for the square root of this variance.

If \(A\) is truly better, so that \(\Delta=\theta_A-\theta_B>0\), the probability that a repeated interim sample ranks \(A\) above \(B\) is the probability that \(D>0\). Standardizing \(D\) gives the key result.

P(\hat{\theta}_A>\hat{\theta}_B) =P(D>0) =P\left(\frac{D-\Delta}{SE_D}>-\frac{\Delta}{SE_D}\right) =\Phi\left(\frac{\Delta}{SE_D}\right),

where \(\Phi\) is the standard normal cumulative distribution function. The probability of reversing the true order is therefore \(1-\Phi(\Delta/SE_D)\), assuming ties have probability zero. The ratio \(\Delta/SE_D\) is the true gap measured in standard errors. A larger gap or a smaller standard error makes correct ordering more likely.

In practice, the true gap \(\Delta\) is unknown. The formula is therefore most straightforward as an operating-characteristic calculation under a stated hypothetical scenario for the true effects. One can also substitute the observed gap for \(\Delta\) to obtain a plug-in repeat-sampling stability calculation: it describes what the model predicts if the current estimates were the true effects. It is not the probability that the observed ranking is true, nor a posterior probability that one arm is better.

Worked Examples: Close and Widely Separated Effects

Worked Example: Close True Effects

Suppose a hypothetical interim comparison concerns a continuous outcome measured in points, with larger effects preferred. Under one planning scenario, the true effect is 5 points for arm \(A\) and 4 points for arm \(B\). Assume unbiased, independent, approximately normal estimates, each with standard error 2 points. These assumptions make the variance of the estimated difference the sum of the two variances.

The true gap is \(\Delta=5-4=1\) point, and \(SE_D=\sqrt{2^2+2^2}=\sqrt{8}=2.828\) points. Thus the probability of observing \(A\) above \(B\) is

P(\hat{\theta}_A>\hat{\theta}_B) =\Phi\left(\frac{1}{2.828}\right) =\Phi(0.3536) \approx 0.638.

Even though \(A\) is better in this hypothetical scenario, an interim sample of this precision ranks it first only about 63.8% of the time. The chance of reversing the order is about \(1-0.638=0.362\), or 36.2%. The calculation illustrates why a small observed difference should not be treated as a stable ordering when standard errors are large relative to the gap.

Worked Example: Widely Separated True Effects

Now suppose a hypothetical scenario has true effects of 8 points for arm \(A\) and 2 points for arm \(B\). Retain the same assumptions and standard errors: each interim estimate is unbiased and approximately normal with standard error 2 points, and the two estimates are independent.

Here \(\Delta=8-2=6\) points and \(SE_D=\sqrt{2^2+2^2}=2.828\) points. Therefore,

P(\hat{\theta}_A>\hat{\theta}_B) =\Phi\left(\frac{6}{2.828}\right) =\Phi(2.121) \approx 0.983.

Under this scenario, the interim ranking is correct about 98.3% of the time, and the reversal probability is about 1.7%. The standard errors have not changed from the preceding example; what changed is the separation of the true effects. Ranking reliability is governed by the gap relative to the uncertainty, not by the gap alone.

Reading a Ranking With Three Arms

With several candidate arms, pairwise calculations help identify which parts of an ordering are fragile. They do not, by themselves, give the probability that the entire ranking is correct. In particular, comparisons that share an arm are statistically dependent: the estimate for that arm appears in both differences. Multiplying pairwise probabilities as if they were independent would generally be incorrect.

Worked Example: A Close Top Pair and a More Distant Third Arm

At a hypothetical interim, three arms have estimated effects of 6.2, 5.5, and 3.0 points for arms \(A\), \(B\), and \(C\), respectively. For a simple stability check, suppose each estimate has standard error 1.5 points and the estimates are independent. Use the observed estimates as plug-in centers. This is a model-based repeat-sampling calculation, not a probability that the current ranking is the true ranking.

For any pair, the standard error of the difference is \(SE_D=\sqrt{1.5^2+1.5^2}=\sqrt{4.5}=2.121\) points. For \(A\) versus \(B\), the observed gap is \(6.2-5.5=0.7\) points, so the plug-in probability that a repeat sample ranks \(A\) above \(B\) is

\Phi\left(\frac{0.7}{2.121}\right) =\Phi(0.330) \approx 0.629.

The top two estimates are close relative to their uncertainty, so their order is fragile under this model. For \(A\) versus \(C\), the gap is \(6.2-3.0=3.2\) points: \(\Phi(3.2/2.121)=\Phi(1.508)\approx 0.934\). For \(B\) versus \(C\), the gap is \(5.5-3.0=2.5\) points: \(\Phi(2.5/2.121)=\Phi(1.179)\approx 0.881\).

The calculations suggest that the main instability is the ordering of \(A\) and \(B\), not the separation of the third estimate from the first two. They do not establish a 62.9% probability that \(A\) truly exceeds \(B\). Nor can the three pairwise results be multiplied to claim a probability for the full ranking: their comparisons are dependent, and the plug-in centers are not known true effects.

Why the Result Holds, and What to Check

The pairwise result follows directly from the sampling distribution of a difference. If the two estimates are jointly normal, any linear combination of them is normal. In particular, \(D=\hat{\theta}_A-\hat{\theta}_B\) has mean \(\theta_A-\theta_B=\Delta\). Its variance is

\operatorname{Var}(D) =\operatorname{Var}(\hat{\theta}_A)+\operatorname{Var}(\hat{\theta}_B) -2\operatorname{Cov}(\hat{\theta}_A,\hat{\theta}_B) =SE_A^2+SE_B^2-2\operatorname{Cov}(\hat{\theta}_A,\hat{\theta}_B).

Thus \(Z=(D-\Delta)/SE_D\) has a standard normal distribution. The event that \(A\) ranks higher is \(D>0\), which is equivalent to \(Z>-\Delta/SE_D\). By symmetry of the standard normal distribution, \(P(Z>-x)=\Phi(x)\), giving \(P(D>0)=\Phi(\Delta/SE_D)\). This derivation also shows why ignoring covariance can misstate ranking uncertainty.

Before using the calculation, check that the arms are compared using the same endpoint, effect scale, direction, and interim data cutoff. Confirm that estimates are approximately unbiased for the target effects and that standard errors are appropriate at the interim. If estimates share data or a control group, their covariance may not be zero; use the covariance implied by the design and estimator rather than assuming independence. With estimated standard errors, the normal calculation is an approximation, especially when the interim sample is small.

Several warning signs call for caution:

A tiny gap relative to \(SE_D\): the observed order is likely to change under repeated sampling.
Different estimands or scales: ranking unlike quantities—for example, a risk difference against a hazard ratio—has no direct interpretation.
Unequal precision: an arm can have a larger estimated effect but a less precise estimate. Ranking by a standardized statistic would answer a different question.
Strong dependence or an incorrect covariance: this changes the uncertainty of the difference and can materially change the ranking probability.
Unstable endpoint behavior: heavy skew, sparse binary outcomes, or very few events may make a normal approximation unreliable.
Unplanned analysis choices: changing the endpoint, population, or direction after seeing interim results can make the ranking difficult to interpret and reproduce.

What Ranking Can and Cannot Do

A point-estimate ranking is simple and transparent, but it suppresses uncertainty. The pairwise normal calculation adds a scale for that uncertainty, conditional on its model and assumptions. It is not a substitute for a prespecified selection rule, and it does not say whether an effect is clinically worthwhile. A ranking can be stable while all candidate effects are too small to matter; conversely, a potentially useful arm can be hard to distinguish from another when interim data are limited.

Other approaches answer related but different questions. Ranking by a \(z\)-statistic or \(p\)-value emphasizes evidence relative to a reference value and can reorder arms with unequal standard errors. Confidence intervals describe uncertainty in individual effects or contrasts, but overlap or non-overlap of separate intervals is not itself a direct calculation of ranking probability. A Bayesian analysis can calculate a probability of one effect exceeding another, but that probability depends on the likelihood and prior distribution. Each approach requires its own assumptions and should be aligned with the trial’s objective.

For more than two arms, a full ranking-stability analysis must account for the joint distribution of all estimates. Pairwise calculations are useful diagnostics, not a complete joint answer. They can reveal that a top pair is nearly tied in precision units while another arm is well separated. The interpretation should remain conditional on the sampling model, the interim design, and the chosen effect scale.

Common Mistake

Common mistake: “Arm A’s interim estimate is larger, so A is probably the truly better arm.” This treats the observed ordering as if it were a direct probability statement about the unknown effects. The correction is to distinguish the observed order from the true order and quantify repeated-sampling ranking stability using the effect gap and the standard error of the difference. A plug-in stability probability is not a posterior probability that the current winner is truly best.

Key takeaway: An interim ranking is most fragile when the differences between estimates are small relative to their sampling uncertainty. Report the ordering together with an uncertainty assessment, state the model and covariance assumptions, and avoid interpreting a rank as certainty about the true effects.

Check Your Understanding

For each question, state the reasoning as well as the numerical result where relevant. Solutions are provided below.

Two arms have a hypothetical true-effect gap of 3 units. Their unbiased independent interim estimates each have standard error 2 units. What is the probability that the truly better arm ranks higher?
For the same pair of arms, what happens to the ranking probability if both standard errors are reduced to 1 unit?
Why is it generally wrong to multiply pairwise ranking probabilities to obtain the probability that a three-arm ranking is entirely correct?
An analyst reports a plug-in probability of 0.80 that arm \(A\) will rank above arm \(B\) in a repeat sample. Does this mean there is an 80% probability that \(A\) truly has the larger effect?

Solutions. (1) The difference has standard error \(\sqrt{2^2+2^2}=2.828\) units, so the probability is \(\Phi(3/2.828)=\Phi(1.061)\approx 0.856\). (2) The difference standard error becomes \(\sqrt{1^2+1^2}=1.414\) units, giving \(\Phi(3/1.414)=\Phi(2.121)\approx 0.983\); better precision makes the correct order more stable. (3) Pairwise comparisons share estimates and are therefore dependent; their probabilities cannot generally be multiplied as if independent. (4) No. It is a repeat-sampling probability under the stated plug-in model, not a probability about the unknown true effects. A posterior probability would require a Bayesian model, including a prior.