Operating Characteristics of Intermediate-Based Rules

Why Evaluate a Rule Across Effect Scenarios?

An intermediate-based rule is a decision made before the final outcome is fully available: an arm is retained for further study if its intermediate result meets a prespecified criterion. A threshold may be calibrated to retain promising arms, but its practical performance also depends on how intermediate effects relate to final effects. An arm can have a strong intermediate result and a weak final effect, or the reverse. A useful evaluation therefore asks not only how often an arm is retained, but also how often a clinically worthwhile arm is dropped and how often a retained arm ultimately meets the final success criterion.

These probabilities are operating characteristics: they describe the long-run behavior of a specified rule under stated true-effect scenarios. They are not guarantees about an individual arm. In particular, a retention probability is not the probability that the treatment is effective, and a false-drop probability is defined relative to a stated clinical classification of the arm.

A Joint Model for Intermediate and Final Evidence

Consider one candidate arm. Let \(Z_I\) and \(Z_F\) be its standardized statistics for the intermediate and final outcomes, oriented so that larger values favor treatment. For an illustrative fixed-look evaluation, assume

Here \(\eta_I\) and \(\eta_F\) are the means of the standardized statistics under the chosen true-effect scenario, and \(\rho\) is their correlation. The unit variances and joint normality are modeling assumptions, not automatic properties of every time-to-event analysis. They may be reasonable approximations for suitable large-sample statistics, but should be checked against the trial’s analysis plan and design. A more detailed model may be needed when there are substantial departures from normality, unequal variances, or complex dependence.

Suppose the arm is retained when \(Z_I\ge c_I\), and final success is declared when \(Z_F\ge c_F\). These cutoffs are assumed fixed for the evaluation. The final cutoff should be the one specified by the design; if it incorporates multiplicity control, that calibration must be reflected in \(c_F\). This single-arm calculation does not itself establish familywise error control across several arms.

where \(\phi\) is the standard normal density. Dividing this joint probability by the retention probability gives the probability of final success among retained arms, when the retention probability is positive. These calculations use the conditional distribution \(Z_F\mid Z_I=x\sim N\big(\eta_F+\rho(x-\eta_I),1-\rho^2\big)\).

Worked Example: Retention, False Drop, and Joint Success

Why Correlation Changes Post-Selection Success

When the intermediate and final statistics are correlated, a retained arm is not a random draw from all arms under that scenario. For positive \(\rho\), an intermediate statistic above its mean tends to accompany a final statistic above its own mean. The final-success probability among retained arms can then exceed the marginal final-success probability. Negative correlation can produce the opposite pattern. The marginal probability \(\Phi(\eta_F-c_F)\) does not change with \(\rho\), but the joint and conditional probabilities generally do.

A Short Justification for the Joint Calculation

The joint expression follows by conditioning on the observed intermediate statistic. Under the stated bivariate normal model, the conditional distribution of the final statistic is normal with mean \(\eta_F+\rho(x-\eta_I)\) and variance \(1-\rho^2\). Hence, at \(Z_I=x\), the probability of final success is

The density of \(Z_I=x\) is \(\phi(x-\eta_I)\). Multiplying the conditional success probability by that density and integrating over the retained region \(x\ge c_I\) yields the joint probability. If \(\rho=0\), the conditional probability does not depend on \(x\); the integral factors into the product of the marginal retention and final-success probabilities. This establishes why multiplying the marginals is valid under independence, but generally not under dependence.

For the special case in the preceding example, the centered bivariate normal quadrant probability has derivative \(1/(2\pi\sqrt{1-\rho^2})\) with respect to \(\rho\), and equals \(1/4\) at \(\rho=0\). Integrating that derivative from \(0\) to \(\rho\) gives \(1/4+\arcsin(\rho)/(2\pi)\).

Evaluate Clinically Relevant Effect Combinations

Operating characteristics should be evaluated over combinations that represent the scientific questions the design is meant to address. For instance, a clinically worthwhile arm may have a strong intermediate effect and a modest final effect, or a weak intermediate effect despite a worthwhile final effect. The latter combination is particularly informative about false dropping. The scenario classification must be stated in terms of the true final effect and the prespecified clinical target; passing a statistical final threshold is a separate event.

Checks, Alternatives, and Limitations

A useful evaluation reports the scenario, the model, the thresholds, and which probability is being summarized. For each clinically relevant combination, check the following:

• Retention and false-drop probabilities for a fixed scenario must sum to one.
• The joint probability of retention and final success cannot exceed either marginal probability.
• Under independence, joint success must equal retention multiplied by marginal final success.
• Increasing \(\eta_I\) while holding the threshold fixed increases retention; increasing \(\eta_F\) increases marginal final success.
• With dependence, the conditional final-success probability among retained arms need not equal marginal final power.

A common mistake is to report \(P(Z_F\ge c_F)\) as the probability of final success after the selection rule. That is only the marginal final-success probability. If the reported quantity requires both retention and final success, use the joint probability; if it is success among retained arms, divide the joint probability by retention. Another mistake is to multiply marginal probabilities without checking independence.

The principal alternative to a scenario grid is to report one or two prespecified representative scenarios. That can be concise, but may hide poor performance when intermediate and final effects disagree. A grid makes trade-offs visible, while a model-based evaluation with nonzero \(\rho\) addresses dependence that an independence calculation omits. Neither approach resolves uncertainty about the intermediate–final relationship: the assumed means and correlation are inputs, not facts learned from the operating-characteristic calculation. Likewise, these single-arm quantities do not summarize all-arm selection, multiplicity, resource use, or the trial’s overall error rates.

Unexpected values are a reason to inspect the inputs and the calculation rather than to interpret the result immediately. In particular, verify that both statistics are oriented in the beneficial direction, that the thresholds correspond to the actual rule, that \(\rho\) is on the appropriate standardized scale, and that rounding has not changed a reported result. If a normal approximation is doubtful, or the relationship varies substantially across scenarios, the assumed model may not adequately represent the design.