Why Missing Intermediate Outcomes Matter at Selection
At a multi-arm, multi-stage (MAMS) selection look, intermediate outcomes can help determine which treatment arms continue. If an outcome is unavailable for some participants, the observed participants may not represent the randomized arm. A selection contrast calculated from them can therefore differ from the contrast that would have been calculated had the outcomes been observed. Since an arm may be retained or dropped based on that contrast, handling missingness is part of the selection rule—not just a data-cleaning choice.
This tutorial compares three ways to estimate arm-level intermediate outcomes: complete-case analysis, likelihood-based analysis, and inverse-probability weighting. The examples use a continuous intermediate outcome measured in days, with larger values indicating a more favorable result. The same reasoning applies to other outcomes when the model and estimand are appropriate. A value that is not yet available at the look is not automatically a missing value: pending outcomes and delayed ascertainment require their own treatment, as covered in the earlier material on delayed intermediate outcomes.
The contrast's direction must be defined in advance. Here, positive values favor the experimental arm. If a shorter time or lower score is favorable, the outcome or contrast direction must be oriented accordingly. The selection rule also needs to be prespecified; an illustrative rule used below retains an arm when its estimated contrast is at least 2 days. That cutoff is only an example, not a generally calibrated decision threshold.
Missingness Assumptions and Three Approaches
The key distinction is how outcome availability relates to the outcome itself and to information observed for everyone. Missing completely at random (MCAR) means availability is independent of both the outcome and relevant observed characteristics, within the randomized arm. Missing at random (MAR) means availability may depend on observed information, but after conditioning on that information it does not additionally depend on the unobserved outcome. Missing not at random (MNAR) means availability can still depend on the unobserved outcome after conditioning on observed information.
A complete-case estimate uses only participants with \(R_i=1\). It is simple and can be unbiased for an arm mean under MCAR, subject to ordinary sampling assumptions. But if outcome availability depends on observed characteristics that also predict the outcome, the observed participants may have a different covariate mix from the randomized arm. An unadjusted complete-case mean then generally targets the mean among observed participants, not \(\mu_a\).
A likelihood-based approach specifies a probability model for \(Y_i\) given treatment arm and observed information, and estimates its parameters using the observed outcomes. Under MAR, a correctly specified outcome model, and the usual conditions that permit the missingness mechanism to be ignored for outcome-model inference, this can estimate the arm mean. For example, the model can estimate outcome means within baseline-covariate groups and then average those predictions over the randomized arm's covariate distribution.
An inverse-probability-weighted (IPW) approach weights observed outcomes by the inverse of their probability of being observed. Participants who resemble people less likely to have observed outcomes receive more weight. Under MAR and positivity, with correctly specified observation probabilities, the weighting can recover the arm mean. Estimated probabilities are often used in practice; their estimation adds uncertainty and makes correct specification and diagnostics important.
Why the Methods Behave Differently
Under MCAR within an arm, complete cases are, in expectation, a random subset of that arm. Under MAR, however, the probability of appearing in the complete-case sample can vary with \(X_i\). If \(X_i\) also predicts \(Y_i\), the distribution of \(X_i\) among complete cases differs from its distribution in the randomized arm. That difference can change the mean even when, within each \(X_i\) group, missingness does not depend on the unobserved outcome.
The likelihood and IPW approaches use observed \(X_i\) differently. A likelihood model predicts outcomes conditional on \(X_i\), then standardizes those predictions to the target arm. IPW instead reweights observed outcomes so that, under its assumptions, the weighted participants represent the target arm. Neither method removes the need for assumptions: likelihood depends on the outcome model, while IPW depends on the observation model.
The equality follows under MAR: conditional on arm and \(X_i\), \(R_i\) is independent of \(Y_i\), and \(E(R_i\mid Z_i=a,X_i)=\pi_a(X_i)\). Averaging this identity over the target covariate distribution gives the arm mean. In a finite sample, the usual normalized IPW estimate divides the weighted outcome total by the total weight; it is a ratio estimate, so it need not equal the population identity exactly.
For a likelihood analysis, suppose the outcome model is \(f_\theta(y\mid Z_i,X_i)\), and the missingness model depends on observed \(Z_i,X_i\), not on an unobserved \(Y_i\) after conditioning on them. The contribution to the observed-data likelihood for a participant with \(R_i=1\) includes \(f_\theta(Y_i\mid Z_i,X_i)\); for \(R_i=0\), the outcome contribution is integrated over all possible values and equals 1. Under MAR and the relevant parameter-separation conditions, the missingness factor can be omitted when estimating \(\theta\). A wrong outcome model or an MNAR mechanism can invalidate this argument.
Worked Examples on the Same Hypothetical Data
The following examples use hypothetical data from one control arm and one experimental arm. Every participant has an observed baseline category, \(X\), classified as low or high. There are 10 participants in each category in each arm. The table reports, for each arm and category, the number with observed outcomes and the sum of those observed outcomes.
| Arm | Baseline category | Randomized | Outcomes observed | Observed outcome sum (days) | Observed mean (days) |
|---|---|---|---|---|---|
| Control | Low | 10 | 8 | 32 | 4 |
| Control | High | 10 | 4 | 32 | 8 |
| Experimental | Low | 10 | 4 | 20 | 5 |
| Experimental | High | 10 | 8 | 80 | 10 |
For these examples, assume that within each arm and baseline category, observed outcomes are representative of that category's outcomes. This is a MAR assumption conditional on arm and category. Also assume that the table's cell means are suitable estimates of the corresponding category-specific means. The numbers are illustrative; the table alone cannot verify these assumptions.
Worked Example: Complete-Case Selection Contrast
Calculate the unadjusted complete-case mean in each arm and apply the illustrative rule that retains the experimental arm if its estimated contrast is at least 2 days.
The control complete-case mean combines 32 days from 8 low-category participants and 32 days from 4 high-category participants:
The experimental complete-case mean combines 20 days from 4 low-category participants and 80 days from 8 high-category participants:
Thus, \(\hat{\Delta}_{1,\mathrm{CC}}=8.333-5.333=3.000\) days. It meets the illustrative 2-day cutoff, so this complete-case estimate would retain the arm. But the observed groups have different category mixes: 8 of 12 control participants are in the low category, compared with 4 of 12 experimental participants. Because the categories have different outcome means, this unadjusted contrast does not correct for that imbalance.
Worked Example: Likelihood-Based Standardization
Fit a simple normal outcome model with separate means for each arm-by-category cell. Assume outcomes within a cell are independent and normally distributed with a cell-specific mean and variance, and that observation is MAR conditional on arm and category. For this cell-means model, the likelihood estimate of each cell mean is the observed cell mean. Standardize those estimates to the randomized arm distribution, which has 10 low- and 10 high-category participants in each arm.
The standardized control mean is the average of its two cell means, weighted equally because both categories have 10 randomized participants:
The standardized experimental mean is:
The likelihood-based contrast is \(7.500-6.000=1.500\) days, below the illustrative cutoff, so this estimate would not retain the arm under that rule. This calculation shows how standardizing to the randomized covariate mix differs from averaging over the complete-case mix. A real likelihood analysis would also account for uncertainty in the estimated cell means and in the resulting contrast.
Worked Example: Inverse-Probability-Weighted Contrast
Estimate the observation probability within each arm and category using the observed fraction: \(8/10\) for low-category control participants, \(4/10\) for high-category controls, \(4/10\) for low-category experimental participants, and \(8/10\) for high-category experimental participants. Use normalized inverse-probability weights.
In the control arm, the low-category weight is \(1/0.8=1.25\), and the high-category weight is \(1/0.4=2.5\). The weighted outcome total is \(32(1.25)+32(2.5)=40+80=120\) days. The total weight is \(8(1.25)+4(2.5)=10+10=20\), so:
In the experimental arm, the low-category weight is \(1/0.4=2.5\), and the high-category weight is \(1/0.8=1.25\). The weighted outcome total is \(20(2.5)+80(1.25)=50+100=150\) days. The total weight is \(4(2.5)+8(1.25)=10+10=20\), giving:
The IPW contrast is \(7.500-6.000=1.500\) days, also below the illustrative cutoff. Here, the standardized likelihood and IPW point estimates agree because both use the same cell-specific information and the same target category proportions. They need not agree in general, especially with different model specifications or estimated probabilities.
Comparison, Diagnostics, and Limitations
| Approach | What it uses | Key conditions for the target arm mean | Common vulnerability |
|---|---|---|---|
| Complete case | Observed outcomes only, without missingness adjustment | MCAR within arm for the unadjusted mean, or an appropriate adjustment for observed predictors | Biased contrast when observed covariate mixes differ and covariates predict outcomes |
| Likelihood-based | An outcome model fitted to observed outcomes, with predictions standardized to the target arm | MAR given included information, a suitable outcome model, and valid likelihood assumptions | Bias or misleading precision if the outcome model is misspecified or missingness is MNAR |
| IPW | Observed outcomes weighted by estimated inverse observation probabilities | MAR given included information, positivity, and a suitable observation-probability model | Unstable estimates from very small probabilities or misspecified weights |
Useful diagnostics begin with a description of availability: report observed and missing counts by treatment arm and by important baseline or observed-history groups. Compare those groups on variables observed for everyone. Large differences do not prove that an estimator is biased, but they warn against assuming that complete cases represent the randomized arm. For IPW, inspect the estimated observation probabilities and the resulting weights; very large weights indicate that some observed participants are standing in for many participants. For a likelihood model, assess whether the chosen outcome model is plausible, including its covariate and treatment structure.
The methods are not interchangeable guarantees. Complete-case analysis is easiest to interpret when MCAR is credible, but can be misleading under MAR. Likelihood can be efficient when its model is adequate and naturally supports outcome prediction and standardization. IPW makes the observation model explicit and can be attractive when that model is more credible than a detailed outcome distribution, but poor positivity can make its estimates highly variable. A prespecified primary approach with sensitivity analyses is often more informative than choosing whichever method gives the preferred selection result.
All three approaches rely on assumptions that cannot generally be confirmed from observed outcomes alone. If missingness depends on the unobserved outcome even after conditioning on measured information, MAR-based likelihood and IPW analyses may be biased. Sensitivity analysis can examine how conclusions change under plausible departures from MAR; it does not make the missing values known. The complete-case estimate can also be reported as a transparent comparison, but agreement with another method does not prove the missingness assumptions.
Common Mistake
A frequent mistake is to say, “The missing outcomes are only 20%, so the complete-case contrast should be close enough.” The percentage missing does not determine the direction or size of bias. In the hypothetical data, the control and experimental complete-case groups have different baseline-category mixes, and the category means differ. As a result, the complete-case contrast is 3.000 days, whereas both MAR-based estimates are 1.500 days. The correction is to examine how availability relates to observed outcome predictors, define the target arm distribution, and justify the missingness and model assumptions—not to judge reliability by the missing fraction alone.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In one arm, outcomes are observed for 6 of 10 low-category participants and 9 of 10 high-category participants. Which assumption would make an unadjusted complete-case mean representative without further adjustment?
- Suppose, under MAR, the estimated observation probabilities are 0.5 for one observed group and 0.9 for another. What are their inverse-probability weights, and which group receives the larger weight?
- Why can a correctly specified likelihood model and IPW produce the same point estimate in the worked examples?
- What is the main concern if some relevant covariate patterns have estimated observation probabilities close to zero?
Solutions. (1) MCAR within the arm makes observation independent of both the outcome and the covariate category, so the observed participants represent the arm in expectation. If availability depends on category, an unadjusted complete-case mean can instead reflect a changed category mix. (2) The weights are \(1/0.5=2\) and \(1/0.9=1.111\), respectively; the group with observation probability 0.5 receives the larger weight because it is less likely to be observed. (3) In the examples, both methods use the same observed cell means and target the same 50:50 category mix. Likelihood standardizes predicted cell means, while IPW reweights observed outcomes to that mix. (4) Positivity may be practically violated: a few observed participants receive extreme weights, making the IPW estimate unstable and highly sensitive to individual observations.