Ranking Arms With Uncertain Intermediate Effects

Why an Intermediate-Outcome Leader May Not Be the Most Certain Leader

At a selection look in a multi-arm multi-stage (MAMS) trial, arms may differ in their estimated intermediate effects even when those estimates are imprecise. Sorting arms by their estimates gives a simple ranking, but the apparent leader can owe its position to sampling variation. A ranking that accounts for uncertainty can look different: an arm with a smaller estimate but substantially greater precision may have a stronger lower confidence bound. Conversely, the data may not support a confident ordering at all.

This tutorial compares rankings based on point estimates with rankings based on confidence intervals. The goal is to describe what the intermediate data say about relative effects, not to assign posterior probabilities to arms or to decide which arms to retain. The introductory material on regression prediction for arm selection concerns predicted effects; here the focus is how uncertainty in estimated intermediate effects changes their ranking.

For example, an effect may be defined as the difference between an arm and control in the probability of being event-free at a prespecified landmark. In that case, orient the contrast as arm minus control, so positive percentage-point effects are favorable. If the original endpoint uses a scale on which smaller values are better, reverse the contrast before ranking. A ranking is meaningful only after the effect direction, scale, population, and time point have been specified.

Assumptions and the Role of Precision

The calculations below use a large-sample normal approximation. For each arm, assume the estimator is approximately unbiased for the specified intermediate effect, its standard error estimates its sampling variability, and the joint distribution of the arm-effect estimates is approximately multivariate normal. A two-sided 95% interval then uses \(z_{0.975}=1.96\). These are working assumptions: sparse events, small samples, or poor outcome maturity can make the approximation unreliable.

Arm estimates are not always independent. When each treatment arm is compared with the same control group, the estimates can be correlated because they share control data. For two arms \(i\) and \(j\), the standard error of their estimated difference is therefore based on the covariance as well as their separate standard errors:

The covariance must be estimated or obtained from the design and analysis model. Treating estimates as independent when they share a control group can produce an incorrect contrast interval. The covariance is not automatically positive in every analysis, so it should not be guessed from the fact that arms share a control.

How Confidence-Based Rankings Can Reverse

A lower-confidence-bound ranking penalizes an estimate for uncertainty. For two arms with equal confidence level, arm \(A\)'s lower bound is below arm \(B\)'s even though \(A\)'s estimate is higher whenever

Rearranging the second inequality gives

Thus, a reversal of this kind requires \(SE_A>SE_B\): arm \(A\)'s estimated advantage must be smaller than the confidence-bound penalty arising from its greater uncertainty. The derivation is simply subtraction of the two lower bounds; it does not imply that the lower-bound ranking is the only reasonable ranking or that it is a formal test of a treatment difference.

A lower-bound ranking gives every arm a score, so it produces a total ordering if there are no ties. Pairwise confidence comparisons answer a different question: whether the data support a direction for a particular contrast. They can leave some pairs unordered. Neither approach converts a confidence interval into a probability that a parameter lies in a particular range.

Pairwise Contrasts and Shared-Control Covariance

When the Data Do Not Establish an Ordering

Choosing and Interpreting a Ranking

The estimate-based ranking is useful as a concise description of the observed data. The lower-bound ranking can show how a chosen uncertainty penalty changes that ordering, provided the confidence level and effect scale are stated. Pairwise intervals are most direct when the question is whether one specified arm's intermediate effect exceeds another's. For many arms, pairwise claims require care: a collection of marginal intervals is not automatically a simultaneous confidence statement about all pairs.

Diagnostics matter before interpreting any of these rankings. Check whether the standard errors reflect the actual interim analysis, including shared control, censoring, and the available maturity of event data. Confirm that all arms use the same endpoint definition and landmark, and that larger values have been oriented consistently as favorable. A ranking that changes sharply under modest changes to the confidence level or standard-error estimates is fragile. Very wide intervals, sparse events, or unstable estimates are warnings that fine ordering is not informative.

A common mistake is to say, “A has the higher estimate and its interval overlaps B's, so A is not different from B,” or to reverse this and treat non-overlap of marginal intervals as the required pairwise test. Neither argument correctly evaluates the difference. Calculate an interval for \(\theta_A-\theta_B\), using its covariance. Also avoid interpreting a lower confidence bound as a probability statement: a 95% procedure has its stated repeated-sampling coverage under its assumptions; it does not mean there is a 95% probability that this fixed parameter exceeds the observed bound.

Finally, an intermediate-outcome ranking is not automatically a ranking of final time-to-event benefit. Its relevance depends on the relationship between the intermediate and final outcomes and on the analysis plan. Confidence-based ordering describes uncertainty about the specified intermediate effects; it does not establish final-endpoint success, and it does not by itself prescribe an arm-retention rule.