Common Errors in Intermediate-Based Arm Selection

Why Intermediate-Based Selection Needs Care

In a time-to-event MAMS design, an intermediate outcome can be available sooner than the final event endpoint. It may therefore help decide which treatment arms continue. But an early measure is not automatically a safe substitute for the endpoint that ultimately matters. A selection rule can be executed exactly as written and still make poor choices if its assumptions are wrong, its data are immature, or its uncertainty is calculated as though comparisons were independent.

This tutorial focuses on four recurring errors: treating prognostic association as surrogacy, ignoring outcome maturity, choosing a selection threshold after inspecting the results, and neglecting dependence induced by a shared control group. Throughout, let arm \(0\) denote control and arms \(j=1,\ldots,K\) denote experimental treatments. Let \(S\) be an intermediate outcome and \(T\) a time-to-event endpoint. Suppose larger \(S\) is favorable; if the intermediate is instead a time or score for which smaller values are favorable, the direction of every comparison must be reversed consistently.

Definition: A prognostic association means that the intermediate outcome \(S\) is associated with the later endpoint \(T\) among individuals or groups. Surrogacy is a stronger claim: treatment effects on \(S\) provide reliable information about treatment effects on \(T\) for the treatments and population of interest. An association between \(S\) and \(T\) alone does not establish that changing \(S\) through treatment will predictably change \(T\).

Prognostic Does Not Mean Surrogate

A marker can identify participants with a better prognosis without capturing how treatment affects the final endpoint. For example, people with a favorable biomarker may tend to live longer in every arm. That is a prognostic relationship. Yet a treatment could improve the biomarker through a pathway that does not improve survival, or could improve survival without changing that biomarker. In either case, a rule that advances arms solely because they improve the marker can mis-rank treatments for the final endpoint.

In potential-outcome terms, write \(S(j)\) and \(T(j)\) for the intermediate and final outcomes that a participant would have under arm \(j\). A relationship between \(S(j)\) and \(T(j)\), or between observed \(S\) and \(T\), does not by itself show that the treatment contrasts in \(S\) track the treatment contrasts in \(T\). The unobserved counterfactual outcomes prevent a simple within-person association from proving that relationship. Surrogacy is therefore a design justification requiring evidence beyond a correlation or a prognostic regression.

This distinction matters directly for selection. A prespecified rule such as “retain any arm whose response rate exceeds control by at least 10 percentage points” is a rule about \(S\). Its usefulness for a survival objective depends on how reliably that criterion identifies arms with acceptable effects on \(T\). Earlier material on sensitivity analysis for the intermediate-final relationship and operating characteristics of intermediate-based rules addresses how to evaluate that dependence under stated assumptions. The error is to act as if the relationship were guaranteed merely because people with favorable intermediate values tend to have favorable outcomes.

Worked Example: A Prognostic Biomarker Mis-ranks Treatments

Consider a hypothetical trial with 200 participants assigned to each of control, treatment A, and treatment B. At an interim assessment, 60 control participants, 130 participants on A, and 120 on B have a favorable biomarker response. By a prespecified follow-up time, the event endpoint has occurred in 80 control participants, 90 on A, and 60 on B. These are illustrative observed proportions, not claims about sampling precision.

The favorable biomarker response proportions are \(60/200=0.30\) for control, \(130/200=0.65\) for A, and \(120/200=0.60\) for B. Relative to control, A’s observed response difference is \(0.65-0.30=0.35\), or 35 percentage points; B’s is \(0.60-0.30=0.30\), or 30 percentage points. A rule based only on biomarker response would rank A ahead of B.

The observed event proportions are \(80/200=0.40\) for control, \(90/200=0.45\) for A, and \(60/200=0.30\) for B. Thus, in these hypothetical data, A has a 5-percentage-point higher event proportion than control, whereas B has a 10-percentage-point lower event proportion. The biomarker-based ranking and event-based ranking disagree.

This small example does not establish that A harms survival or that B improves it; uncertainty and time-to-event analysis still matter. It does show why a high response rate is not, by itself, proof of benefit on the event endpoint. A biomarker may be prognostic while the treatment-induced changes in it fail to represent treatment effects on the final endpoint.

Immature Outcomes Can Change the Ranking

An interim value is mature only when it has had the opportunity to be observed according to the outcome’s definition and assessment window. For a binary response defined at six months, a participant enrolled two months ago has not yet had the full six-month opportunity. For survival, follow-up may be incomplete and event counts may be too small for a stable comparison. Treating “not yet observed” as “no response” or “no event” confuses missing future information with an observed outcome.

A simple comparison of observed cases can also be misleading if the arms differ in follow-up time or enrollment timing. The observed participants may not represent the full randomized groups: those with mature assessments may be earlier enrollees, and their outcomes can differ for reasons related to calendar time, prognosis, or assessment availability. Before applying a selection rule, state its estimand (for example, response by six months among all randomized participants) and verify that the interim data support estimation of that quantity. If they do not, the rule needs a prespecified method for incomplete follow-up or a later decision time.

Worked Example: Unequal Maturity Reverses an Interim Comparison

Suppose 60 participants have been assigned to each of two hypothetical arms. The intermediate endpoint is response by six months. At the interim look, only 30 participants in arm A have completed six months of follow-up, and 21 of those have responded. In control, 54 participants have completed six months, and 32 have responded.

Among participants with a completed assessment, A’s response proportion is \(21/30=0.70\), or 70%. Control’s is \(32/54\approx0.5926\), or 59.3%. On this observed-mature subset, A is ahead by about \(0.70-0.5926=0.1074\), or 10.7 percentage points.

If the 30 not-yet-mature participants in A are incorrectly counted as nonresponders, the apparent A proportion becomes \(21/60=0.35\), or 35%. If the six not-yet-mature control participants are similarly counted as nonresponders, control becomes \(32/60\approx0.5333\), or 53.3%. This incorrect calculation reverses the ranking.

Neither calculation automatically estimates the intended six-month response proportion for all randomized participants: the completed-case comparison can be biased if maturity is associated with outcome, and the calculation that labels immature participants as failures is wrong by definition. The appropriate response is to follow a prespecified interim estimand and missing-follow-up method, or to delay the decision until sufficient assessments have matured. The arithmetic identifies the problem; it does not repair it.

Thresholds Must Be Set Before Results Are Seen

A selection threshold determines which observed results are considered strong enough to retain an arm. If the threshold is moved after looking at the data, the actual rule is no longer the one that would have been evaluated in advance. A lower cutoff may be chosen when all arms look weak, or a higher cutoff when one arm looks unusually good. Either way, the chance of an accidental selection under a null or unpromising scenario can differ from the chance calculated for a fixed rule.

The following short derivation illustrates the issue using a deliberately simplified setting. Suppose three independent standardized interim statistics \(Z_1,Z_2,Z_3\) each follow \(N(0,1)\) under a global null, and a fixed one-sided cutoff of 1.645 is used. For each arm, the probability of exceeding that cutoff is approximately 0.05. The probability that none exceeds it is \(0.95^3\), so the probability that at least one does is

P\!\left(\max_{1\le j\le 3} Z_j>1.645\right) =1-P(Z_1\le 1.645,Z_2\le 1.645,Z_3\le 1.645) =1-0.95^3 \approx 0.1426.

Thus, under these simplifying assumptions, a rule retaining any arm above the per-arm 5% cutoff has about a 14.3% chance of retaining at least one arm under the global null. This is not a universal MAMS error rate: actual statistics may be dependent, and selection and final testing may have different error-control objectives. It demonstrates why a per-arm threshold cannot be interpreted as the overall chance of a false selection, and why data-driven threshold changes need to be included in the design’s operating-characteristic evaluation.

Worked Example: A Post Hoc Cutoff Changes the Rule

In a hypothetical study, three arm-specific interim statistics are \(1.2\), \(1.7\), and \(2.0\). The protocol’s prespecified one-sided selection cutoff is \(1.8\). Applying the stated rule, only the third arm is selected because \(2.0>1.8\), while \(1.2\) and \(1.7\) do not exceed the cutoff.

Suppose investigators instead lower the cutoff to 1.6 after seeing that the second statistic is 1.7. Now both the second and third arms are selected. This is not an application of the original rule: the threshold itself was chosen using the observed results. The selection probability associated with the modified procedure cannot be justified by quoting the operating characteristics of the fixed 1.8 cutoff.

A valid design response is to prespecify the cutoff and evaluate the complete selection procedure, including its treatment of multiple arms and any subsequent testing. If a threshold must be adapted, the adaptation rule itself must be specified and its operating characteristics assessed as part of the procedure—not improvised after results are available.

Shared Controls Make Comparisons Dependent

When two or more experimental arms are compared with the same control group, their estimated treatment-control contrasts share the control estimate. Even if participants in different randomized arms are independent, the contrasts are not independent. This matters for joint selection probabilities, the chance that at least one arm crosses a boundary, and uncertainty calculations that use several contrasts together.

The dependence can be derived in a simple model. Assume independent observations across randomized groups, equal group sizes \(n\), and a common variance \(\sigma^2\) for a continuous endpoint. For two treatment arms, define \(D_1=\bar X_1-\bar X_0\) and \(D_2=\bar X_2-\bar X_0\). Each contrast has variance \(2\sigma^2/n\). Their covariance is \(\sigma^2/n\), because the shared control mean appears with a minus sign in both contrasts. Therefore,

\operatorname{Cov}(D_1,D_2)=\operatorname{Var}(\bar X_0)=\frac{\sigma^2}{n}, \qquad \operatorname{Corr}(D_1,D_2) =\frac{\sigma^2/n}{\sqrt{(2\sigma^2/n)(2\sigma^2/n)}}=\frac{1}{2}.

This correlation of one-half applies to the stated equal-variance, equal-size continuous-outcome model. It is not a universal correlation for time-to-event statistics or unequal allocation. For those settings, the joint distribution must reflect the actual design and analysis. The general lesson is that pretending shared-control contrasts are independent can misstate the probability of a joint event, such as selecting at least one arm or selecting several arms.

Worked Example: Recognizing Shared-Control Covariance

Suppose a hypothetical interim continuous biomarker is measured on 100 participants in each of three independent randomized groups: control, A, and B. Assume the endpoint variance is \(16\) units squared in every group. The standard error for either treatment-control difference is

SE(D_j)=\sqrt{\frac{16}{100}+\frac{16}{100}} =\sqrt{0.32}\approx 0.5657\ \text{units}.

The covariance between the A-control and B-control differences is the variance of the shared control mean:

\operatorname{Cov}(D_A,D_B)=\frac{16}{100}=0.16\ \text{units}^2.

Consequently, their correlation is \(0.16/(0.5657\times0.5657)=0.16/0.32=0.50\). A calculation that treats these contrasts as independent would omit this shared-control contribution. Any joint selection calculation should use a dependence structure appropriate to the endpoint and allocation, rather than importing independence simply because the treatment groups themselves are distinct.

Diagnostics, Alternatives, and Limits

These errors are easier to prevent when the decision rule is treated as a complete statistical procedure rather than a single cutoff. Before a look, check what is measured, when it is considered observed, which direction is favorable, how the threshold was set, and which comparisons share data. In simulation or other operating-characteristic calculations, include the assumed relationship between intermediate and final outcomes, the actual follow-up and maturity pattern, the prespecified decision rule, and the covariance induced by shared controls.

Possible surrogate failure: treatment arms show intermediate effects that do not align with the final endpoint in sensitivity analyses or relevant prior evidence. Do not infer surrogacy from a prognostic association alone.
Possible maturity failure: arms have different proportions with completed assessment windows or substantially different follow-up. Report maturity by arm and avoid coding unobserved outcomes as unfavorable outcomes.
Possible threshold failure: the cutoff, direction, or rule changes after interim results are inspected. Preserve the prespecified rule or evaluate a fully specified adaptation instead.
Possible dependence failure: joint error calculations assume independent treatment-control comparisons despite a common control. Use the joint distribution appropriate to the design.

The main alternative to acting on an immature intermediate is to wait for more mature information, accepting the cost of delay. Another approach is to use an intermediate-based rule only when its relationship to the final endpoint is supported and its uncertainty is incorporated into design evaluation. Neither option removes all limitations: intermediate outcomes can be measured with error, relationships may differ across treatments or populations, event endpoints may remain sparse, and selection rules may perform differently under scenarios not included in planning. Earlier material on familywise error with intermediate-based selection distinguishes selection behavior from final inferential error control; favorable selection performance alone does not guarantee valid final inference.

Key takeaway: An intermediate-based selection decision is only as credible as its endpoint relationship, data maturity, prespecified rule, and joint uncertainty calculation. Prognostic association is not surrogacy; incomplete observation is not a negative outcome; a post hoc cutoff is a changed procedure; and shared controls create dependence.

Check Your Understanding

Answer each question, then compare with the solutions below.

What is the difference between a prognostic association and surrogacy for an intermediate outcome?
For a six-month response endpoint, why is it incorrect to count a participant with only two months of follow-up as a nonresponder at an interim look?
Under the simplified independent-null example with three statistics and a one-sided 1.645 cutoff, what is the chance at least one exceeds the cutoff?
Under the equal-size, equal-variance continuous-outcome model, derive the correlation of two treatment-control contrasts sharing a control group.
Name two features that should be checked before using an intermediate outcome to select arms.

Solutions. (1) Prognostic association means \(S\) is associated with the later endpoint; surrogacy requires treatment effects on \(S\) to provide reliable information about treatment effects on the final endpoint. The first does not establish the second. (2) The participant has not yet had the full assessment window, so response status by six months is not yet observed. Calling the participant a nonresponder substitutes an unobserved outcome with a negative one. (3) The probability is \(1-0.95^3=0.142625\), approximately 14.3%, under the stated independence and null assumptions. (4) Each contrast has variance \(2\sigma^2/n\), and their covariance is \(\sigma^2/n\) from the shared control mean. Hence the correlation is \((\sigma^2/n)/(2\sigma^2/n)=1/2\). (5) Suitable answers include evidence for the intermediate-final treatment-effect relationship, follow-up maturity by arm, whether the threshold was prespecified, and whether shared-control dependence is included in joint calculations.