Handling Missing Interim Outcomes in Selection

Why Missing Interim Outcomes Matter

An interim selection rule acts on the outcomes available at a particular decision time. Some participants assigned to an arm may not yet have a recorded outcome: assessment may be delayed, a visit may be missed, or follow-up may have ended early. If those outcomes are simply omitted or replaced, the resulting arm scores can change—and so can the arms selected to continue.

The central issue is not only how many outcomes are missing, but whether the observed participants represent the participants whose outcomes the selection rule is intended to summarize. Random assignment does not guarantee this after assignment: the chance of having an observed interim outcome may be related to treatment, health status, or the outcome itself. The methods below therefore make different assumptions and answer slightly different questions.

For the examples, assume a continuous outcome measured in points, with higher values better. Let \(\mu_j\) denote the mean outcome for experimental arm \(j\), and let \(\mu_0\) denote the control mean. The interim benefit score is \(\mu_j-\mu_0\), estimated using the available-data method under consideration. An illustrative rule retains an arm if its estimated benefit is at least 5 points. This threshold is only a device for showing how summaries affect selection; in an actual design, the selection rule and its thresholds must be prespecified.

Complete-Case Summaries

For arm \(j\), the complete-case mean is the average among participants whose outcomes are observed. Its difference from the corresponding control complete-case mean is a simple interim contrast. This calculation is transparent, but it describes the observed participants directly. It represents the full randomized arm without bias only under suitable missingness conditions.

A useful distinction is between missing completely at random (MCAR), missing at random conditional on observed information (MAR), and missing not at random (MNAR). Under MCAR, observation is independent of the outcome and relevant participant information. Under MAR, missingness may depend on observed covariates or recorded history, but—conditional on those variables—not on the unobserved outcome itself. Under MNAR, missingness still depends on the outcome after conditioning on observed information. Complete-case means generally need MCAR, or other special conditions, for an unadjusted mean to represent the full arm mean.

Model-Based Summaries

A model-based summary uses recorded information to account for outcomes that are missing. For example, a regression model may predict the interim outcome from treatment arm and baseline covariates measured for everyone. A model-based arm mean can then be obtained by averaging predictions over the randomized participants in that arm. Another implementation is multiple imputation: draw plausible missing outcomes repeatedly from a fitted predictive distribution, calculate the arm contrast in each completed data set, and combine the results while reflecting imputation uncertainty.

For a model-based approach to represent the intended arm means, the model must be adequate and the missingness assumption must be credible. A common justification is MAR conditional on the covariates and observed history included in the model. The model also needs enough overlap: relevant covariate patterns among participants with missing outcomes should be represented among participants with observed outcomes. Predictions cannot reliably recover information for groups that are absent from the observed data.

A single predicted value is not equivalent to observing the missing outcome. Treating predictions as fixed observations and using an ordinary standard error typically ignores uncertainty about the missing values and can make an estimate appear too precise. For selection based on an estimate, the point estimate may be the part of interest, but its uncertainty should still be reported and handled as specified in the design.

Conservative Summaries and Their Limits

When outcomes have known limits, a conservative summary can show how poor an arm's benefit could be under an adverse assignment of the missing values. For a higher-is-better outcome bounded between \(L\) and \(U\), a lower bound on an experimental-minus-control mean difference assigns \(L\) to missing experimental outcomes and \(U\) to missing control outcomes. This is conservative for the contrast because lowering the experimental mean and raising the control mean both reduce the difference.

Such a bound is not a model-based estimate of the likely mean, nor is it automatically a confidence interval. It may be deliberately pessimistic and too wide to guide selection. Its value is sensitivity: it indicates whether an arm would still meet the rule under a specified adverse completion of the missing data. The bounds must reflect genuine outcome limits; choosing convenient limits does not make the conclusion conservative in a meaningful sense.

Justification and Practical Comparison

The distinction between complete cases and a model-based summary can be expressed directly. Let \(Y\) be an outcome and \(R\) its observation indicator. Write \(p_R=P(R=1)\). The complete-case expectation is \(E(Y\mid R=1)=E(YR)/p_R\). Therefore

If observation is independent of the outcome, the covariance is zero and the complete-case mean is centered on the full-population mean. If participants with larger outcomes are more likely to be observed, the covariance can be positive and the complete-case mean can be too high. For arm contrasts, this issue applies within each arm; bias in one or both arm means can change the estimated difference. A model-based method attempts to explain this dependence through measured covariates, while a conservative bound avoids predicting the most likely missing values and instead considers adverse ones.

Diagnostics, Failure Modes, and Interpretation

Before applying any summary, report the number expected and observed in each arm and the timing and reasons for missing assessments where known. Compare observed baseline characteristics and recorded histories between participants with and without interim outcomes, by arm. Such comparisons can reveal patterns that make MCAR implausible, but cannot establish that missingness is ignorable: the unobserved outcomes remain unknown.

For a model-based approach, examine whether the model captures important outcome patterns, whether predictions are plausible on the outcome scale, and whether participants with missing outcomes have covariate values supported by observed data. Compare results across reasonable model specifications and, where relevant, evaluate adverse departures from MAR. If predictions are extreme or depend on extrapolation, the model-based selection should not be treated as secure. For a bound, check that the endpoint values are genuinely possible and understand how much the conclusion depends on assigning missing outcomes to those endpoints.

The approaches are alternatives, not a ladder in which one is always superior. Complete-case analysis may be adequate when missingness is negligible or plausibly unrelated to outcomes. Model-based summaries can be more informative when useful covariates explain missingness and the model is credible. Conservative bounds are useful when decision-makers want a transparent adverse scenario, but can be uninformative when many outcomes are missing or the scale is broad. It is often useful to compare methods as a sensitivity analysis, while designating one primary interim rule in advance.

A selection decision is an operational choice under uncertainty, not proof that a retained arm is effective or that a dropped arm is ineffective. Missing-outcome handling should therefore be specified alongside the interim outcome definition and selection rule. If the method changes after seeing interim results, the resulting decision may be difficult to interpret because the assumptions and rule have been chosen in response to the data.