Selection When Intermediate and Final Effects Disagree

Why Discordant Evidence Needs a Prespecified Rule

At an interim selection look, an experimental arm may appear promising on an intermediate time-to-event outcome while its emerging final-endpoint estimate points toward harm. The reverse can also happen: the intermediate result may be weak or unfavorable even as the final endpoint begins to look beneficial. Neither pattern determines the decision by itself. The choice depends on what the protocol said would count as sufficient evidence before these results were observed.

The distinction matters in a multi-arm, multi-stage (MAMS) trial because dropping an arm is consequential and the endpoints need not move together. Earlier material on the surrogate paradox explains why an intermediate effect is not automatically a reliable stand-in for the final effect. Here the focus is narrower: how to compare the two kinds of evidence at a selection look using a rule that is explicit, reproducible, and not rewritten in response to the observed pattern.

Set the Scale and the Decision Rule First

For a time-to-event endpoint, one possible effect scale is the negative log hazard ratio: \(\theta=-\log(HR)\), where \(HR\) is the hazard ratio for experimental treatment versus control. With this convention, \(HR<1\) gives a positive effect estimate and favors treatment; \(HR>1\) gives a negative estimate. The intermediate and final endpoints must each have their own clearly specified estimand and direction. A hazard ratio is not automatically an adequate summary if the proportional-hazards assumption is unsuitable; in that case, the protocol needs an appropriate alternative effect measure and corresponding decision procedure.

Suppose the estimated effects are \(\hat{\theta}_{I,k}\) and \(\hat{\theta}_{F,k}\), with standard errors \(SE_{I,k}\) and \(SE_{F,k}\). Define standardized scores \(Z_{I,k}=\hat{\theta}_{I,k}/SE_{I,k}\) and \(Z_{F,k}=\hat{\theta}_{F,k}/SE_{F,k}\). Positive scores favor treatment. These scores describe the estimates relative to their uncertainty; they do not make the intermediate and final endpoints interchangeable.

The following illustrative rule makes the order of decisions explicit. It is not a universal MAMS rule: a real protocol must justify and calibrate its thresholds for the trial’s endpoints, timing, and design.

The veto prevents a sufficiently adverse emerging final result from being overridden by an attractive intermediate estimate. The rescue condition allows strong favorable final evidence to matter when intermediate evidence is not strongly adverse. In the remaining cases, the intermediate criterion determines retention. Such a rule makes the intended response to disagreement visible, rather than disguising it in an informal judgment.

Assumptions and a Short Justification

The standardized-score approach relies on estimates and standard errors that are appropriate for the stated estimands. For example, a Cox-model log hazard ratio may be approximately normally distributed with its estimated standard error when the model is suitable and the data provide adequate information. The analysis also requires the prespecified handling of censoring, missing outcomes, and any stratification or covariate adjustment. If those conditions are doubtful, the reported score may not represent the intended evidence.

The rule also assumes that both endpoints’ favorable directions, analysis populations, and selection-look data cutoffs were fixed in advance. The emerging final outcome may have fewer events and greater uncertainty than the intermediate outcome. That difference is reflected in its standard error, but does not make an immature final estimate definitive. Conversely, a favorable intermediate estimate cannot erase what the final estimate says. If the two scores are correlated, the decision can still be calculated arm by arm, but calibration of the rule must account for that dependence.

The basic algebra explains why this is a decision procedure, not a claim that the endpoints agree. For positive standard errors, \(Z_I\geq c_I\) is equivalent to \(\hat{\theta}_I\geq c_I SE_I\), and \(Z_F\leq-h\) is equivalent to \(\hat{\theta}_F\leq-hSE_F\). Thus the cutoffs define explicit regions of the estimate-and-uncertainty space. The precedence rule partitions those regions: the harm veto is checked first; if it does not apply, the rescue or intermediate criterion is checked. Every possible pair of scores therefore yields a determinate retain-or-drop decision, including a discordant pair.

Worked Decisions Under the Illustrative Rule

The examples use hypothetical arm-level estimates. Each assumes that the endpoint definitions and directions were prespecified, the estimates and standard errors are valid for those endpoints, and the illustrative thresholds were fixed before the look. The score calculations show how to apply the rule; they do not establish that these particular cutoffs are suitable for another trial.

Diagnostics, Alternatives, and Limitations

Before acting on a discordant result, check that the inputs to the rule are trustworthy. Confirm the endpoint definitions, treatment-versus-control direction, analysis population, data cutoff, and standard-error calculation. Inspect whether the final endpoint has enough events for the approximation used, and whether censoring or missingness differs in a way that could distort an estimate. Review model fit and the plausibility of proportional hazards if a hazard ratio is being used. A sign reversal caused by inconsistent coding is an implementation error, not a scientific disagreement.

Also check that the rule was applied to every arm in the planned way. Do not switch between hazard ratios and another effect scale because one gives a preferred answer, or select a different data cutoff for an arm whose result is inconvenient. In multi-arm settings, the thresholds and selection procedure need calibration for the design, including the dependence between endpoint estimates and the fact that multiple arms are considered. A standardized score alone does not guarantee acceptable error rates or operating characteristics.

The main alternatives answer different questions. An intermediate-only rule is simpler but can ignore adverse emerging final evidence. A final-only rule avoids relying on the intermediate endpoint for selection, but may be uninformative when final events are sparse. An informal joint judgment can seem flexible, but allows the decision standard to drift after results are known. A prespecified combined score is another option, but it requires a defensible way to scale and weight the endpoints and can conceal a strong adverse signal if the other endpoint receives enough weight. The veto-and-rescue illustration instead makes the role of each endpoint explicit; it is not automatically superior to these alternatives.

No selection rule removes uncertainty about the relationship between an intermediate outcome and a final outcome. A rule can drop an arm that would ultimately have succeeded or retain one that ultimately fails. Its performance depends on the endpoints, event timing, effect patterns, and thresholds. The rule should therefore be chosen for a scientific and design rationale, evaluated before use, and reported with the decision it produces—not adjusted to make a particular arm appear more promising.

Common Mistake

Wrong reasoning: “The intermediate result is statistically convincing, so the adverse final estimate must be ignored,” or, in the opposite direction, “The final estimate is currently favorable, so the intermediate criterion can be relaxed.” Both arguments change the decision standard in response to the observed data.

Correction: Calculate the prespecified endpoint scores and apply the decision hierarchy in its fixed order. Explain which condition determined the result. If the rule seems scientifically inadequate, revise it for a future design or a future trial; do not silently revise it at the current selection look.