From a Selection Idea to an Implementable Protocol
An intermediate-based selection design uses an earlier measure to decide which experimental arms continue, while the final endpoint—often time to an event—requires longer follow-up. A protocol that says only “drop arms with weak early results” leaves critical questions unanswered: which result, assessed when, calculated from which participants, and compared using what rule? Those details affect who is selected and how often the design meets its scientific goals.
The protocol should make the selection process reproducible before outcomes are observed. It should also preserve the distinction between an interim decision and the final treatment-effect analysis. The selected set of arms depends on intermediate data; final-endpoint follow-up and analysis must therefore be specified for selected and discontinued arms, not improvised after selection.
Specify the Measure, Population, and Assessment Schedule
Begin by naming the intermediate outcome and its role. State whether it is binary, continuous, ordinal, or a time-to-event measure; define its measurement scale and direction; and identify the treatment contrast that informs selection. For example, “response” is not a sufficient definition. The protocol should state the response criteria, how a response is confirmed, and whether higher or lower values favor treatment.
Define the population contributing to the selection analysis. Specify the relevant assignment groups, any eligibility requirements for the interim analysis, and how participants are handled if treatment stops, changes, or is not received as planned. A participant should not disappear from the analysis merely because an intermediate assessment was not recorded.
The schedule needs more than a nominal visit date. Specify the target assessment time, allowable window, data-cut date or follow-up duration, and the rule for choosing among repeated or out-of-window assessments. For a time-to-event intermediate measure, state the event definition, censoring date, and information threshold used at the selection look. Describe operational checks for whether outcomes are sufficiently mature before the data cut.
Worked Example: Defining a Binary Intermediate Assessment
Suppose a hypothetical trial compares two experimental arms with a control arm. The protocol defines the intermediate outcome as confirmed tumor response by week 12. A response requires at least one qualifying assessment in the week 10–14 window and confirmation at least 4 weeks later. The selection data cut occurs after all participants assigned before a stated enrollment cutoff have had an opportunity to complete the window and confirmation period.
The protocol must resolve what happens when a participant has an assessment at week 15, has an unconfirmed response, or has no scan in the window. For example, it might define week 15 as outside the primary window, count an unconfirmed response as not confirmed by the cut, and classify no scan as missing rather than as a nonresponse. These are design choices, not consequences of the word “response”; they must be chosen and recorded in advance.
Now suppose, for illustration, that at a review cut there are 120 participants assigned to control and 120 to one experimental arm. A response status is recorded for 84 control participants, of whom 50 responded, and for 96 experimental participants, of whom 62 responded. The observed available-outcome proportions are
Thus the observed difference is 5.06 percentage points in favor of the experimental arm. These denominators do not account for all 120 participants in either group: 36 control and 24 experimental outcomes remain unrecorded. The protocol must distinguish pending assessments from missed visits, withdrawals, and other reasons for unavailable outcomes.
As an explicitly extreme sensitivity assignment, suppose every missing experimental outcome is treated as a nonresponse and every missing control outcome as a response. Then
so the experimental-minus-control difference is \(0.5167-0.7167=-0.2000\), or −20.00 percentage points. This deliberately adverse assignment is a sensitivity analysis, not an automatic replacement for the primary analysis. The example shows why missing counts and the direction of each assignment must be checked explicitly.
Prespecify Missing-Data and Joint-Model Assumptions
For every intermediate outcome, describe how the analysis treats each reason for missingness: pending assessment, missed visit, withdrawal, death before assessment, and administrative data cutoff may have different meanings. State the primary analysis approach and the sensitivity analyses that probe plausible departures from its assumptions. If an analysis uses observed outcomes only, explain the assumptions needed for that analysis to represent the intended comparison; do not imply that participants with missing outcomes are equivalent to those observed.
Because the final endpoint is time-to-event, the protocol must also state how the intermediate and final outcomes are linked for design evaluation. Identify the joint model or data-generating assumptions used to plan the selection rule and evaluate its operating characteristics. These may include the association between intermediate response and final event time, any relevant covariates, treatment effects on both outcomes, event and censoring distributions, and whether the association or treatment effect is allowed to vary by arm or subgroup.
A joint model used for planning is not automatically a valid analysis model for the final treatment effect. State which assumptions are used for simulation and which are required for the final analysis. For the latter, specify the final-endpoint estimand, censoring assumptions, analysis method, and any multiplicity procedure needed for the intended claims. If proportional hazards is assumed for a particular analysis, say so and provide a plan to examine departures; do not let the model name stand in for an assumption statement.
The protocol should include sensitivity scenarios for uncertain intermediate–final relationships. For instance, evaluate weaker and stronger associations than the planning value, and plausible departures in treatment effect or missingness. The earlier material on sensitivity analysis for the intermediate-final relationship motivates this check; the protocol’s job is to name the scenarios and the decision criteria they inform.
Worked Example: Writing a Reproducible Selection Rule
Consider a hypothetical design in which each experimental arm is compared with a concurrent control using the estimated week-12 response proportion. The protocol defines \(\hat p_a\) as the observed proportion in arm \(a\), where \(a=0\) denotes control and \(a=1,2\) denotes experimental arms. It specifies that an experimental arm is retained if its estimated response difference from control is at least 0.05 at the selection look.
Suppose the hypothetical observed proportions are \(\hat p_0=0.58\), \(\hat p_1=0.66\), and \(\hat p_2=0.60\). For arm 1, the difference is \(0.66-0.58=0.08\), which exceeds 0.05; retain arm 1. For arm 2, the difference is \(0.60-0.58=0.02\), which is below 0.05; do not retain arm 2 under this rule.
This calculation specifies the decision only if the protocol has already resolved missing outcomes, assessment timing, minimum data maturity, and what happens if neither or both arms meet the criterion. The threshold is an example, not a universal recommendation. A rule based on a confidence bound, a posterior probability, or a different effect measure is also possible, but it must be stated with the same precision and evaluated for its consequences.
Set Operating-Characteristic Targets and Verify Them
An operating characteristic is a probability or expected quantity describing how a design behaves under a specified data-generating scenario. The protocol should identify the scenarios and targets that justify the selection rule. Relevant targets can include the probability of retaining a truly promising arm, the probability of dropping an arm with a specified worthwhile effect, the probability of selecting any ineffective arm under a null configuration, and expected sample size or event count. Targets should be tied to explicit treatment-effect scenarios and stated in context.
When the decision includes selection followed by final testing, evaluate the complete sequence: generate intermediate and final outcomes under the scenario, apply the prespecified selection rule, then apply the final analysis to the arms that remain. A final analysis calibrated in isolation may not have the intended error rate after data-dependent selection. State the error criterion, such as the probability of at least one false positive among the planned comparisons, and the procedure used to meet it.
Worked Example: Checking a Simulation Target
Suppose a hypothetical design uses 10,000 simulated trials under the global null to estimate the probability of at least one false positive after selection. Assume the protocol target is no greater than 0.025. The simulation produces 238 trials with at least one false positive, giving
The Monte Carlo standard error for this estimated probability is
The estimate is below the target, but simulation uncertainty is not zero. The protocol should state whether the target is assessed using the point estimate, an upper confidence bound, or another prespecified criterion, and should use enough replicates for the decision to be informative. It should also evaluate retention and power under scientifically relevant non-null scenarios. A result under the global null alone cannot show that the rule retains arms with worthwhile effects.
Protect Final-Endpoint Follow-Up
The protocol must say what happens to participants in an arm that is not selected. Stopping recruitment to that arm is not the same as ending follow-up for participants already enrolled. Specify whether they continue scheduled final-endpoint follow-up, how events and censoring are collected, and whether treatment or outcome ascertainment changes after the interim decision. Document the planned event target or follow-up horizon and the procedures for obtaining outcomes after treatment discontinuation, where appropriate for the estimand.
A decision based on an intermediate outcome does not make the final outcome unnecessary. If follow-up is stopped selectively, the final data may be less complete and the interpretation of the planned comparison may change. State how the final-endpoint dataset will be frozen, which arms enter each final comparison, and how the selection history is carried into the final analysis and reporting.
Name the intermediate and final endpoints, measurement scales, event definitions, and treatment-effect directions.
Specify assessment windows, maturity requirements, populations, and handling of late or unavailable outcomes.
Write the selection rule mathematically, including thresholds, ties, minimum information, and what happens if no arm qualifies.
State joint-model scenarios, operating-characteristic targets, simulation criteria, final analysis, and continued event collection.
Diagnostics, Alternatives, and Limitations
Before launch, test whether the specification can be applied to realistic edge cases: a response recorded outside the visit window, a delayed confirmation, an arm with few mature assessments, a tie at the threshold, or no experimental arm meeting the rule. During conduct, monitor data maturity, missingness reasons, deviations from assessment windows, and whether the assumed intermediate–final relationship remains plausible. Such monitoring should not silently change the rule; any permitted adaptation and its consequences must be prespecified.
A threshold on an observed effect estimate is transparent and easy to implement, but it may be unstable with small interim samples and sensitive to missing outcomes. Rules using uncertainty measures can account for precision, but require their own stated model and calibration. Selecting on a directly observed final-endpoint quantity may be less indirect when sufficient events are available, but can delay a decision and may alter the design’s efficiency. No alternative removes the need to define the estimand, missing-data handling, and operating-characteristic targets.
The main limitation is that intermediate-based selection is informative only to the extent that the intermediate measure is measured reliably and predicts the final endpoint sufficiently well for the decision at hand. A strong association in historical or planning data does not guarantee the same association in the trial arms or population. Selection can also create a group of retained arms enriched for favorable interim noise. These risks are reasons to evaluate the full design under varied assumptions, not reasons to treat a selected arm’s interim estimate as a final treatment-effect conclusion.
Check Your Understanding
Answer each question using the protocol principles above; concise calculations are sufficient.
- A protocol says “assess response at about three months.” Name two details needed to make the schedule operational.
- In an interim data cut, 30 of 50 observed participants in an experimental arm respond and 24 of 40 observed control participants respond. Compute the observed experimental-minus-control difference. Can the unrecorded outcomes be called nonresponses automatically?
- A selection rule retains an arm when its estimated response difference is at least 0.04. If the estimate is 0.03, what is the decision under the stated rule, and what additional case should the protocol address?
- Why should the operating-characteristic simulation include the selection step before the final test?
- What should happen to final-endpoint follow-up for participants already enrolled in an arm that is dropped?
Solutions. (1) State a target assessment time and allowable window; also specify the data-cut or maturity rule and how late or repeated assessments are handled. (2) The proportions are \(30/50=0.60\) and \(24/40=0.60\), so the observed difference is \(0.60-0.60=0.00\), or 0 percentage points. No: the protocol must classify why the other outcomes are unavailable and prespecify their analysis. (3) Do not retain the arm because \(0.03<0.04\). The protocol should also address a tie at 0.04, as well as what happens if no arm qualifies. (4) Because the final tests are applied only after a data-dependent selection; omitting selection does not evaluate the probability of false positives or power for the design actually planned. (5) The protocol should specify continued collection of final events and censoring information, unless a justified and prespecified alternative changes follow-up; dropping recruitment alone does not imply stopping follow-up.