Missing Outcomes in Utility-Based Comparisons

Why Missing Outcomes Matter for Utility Comparisons

A utility-based comparison summarizes efficacy and safety outcomes on a common scale. If some outcomes are unobserved, the missingness can change the estimated outcome distribution and therefore the expected utility. The central issue is not simply how many values are absent: it is what assumptions connect the unobserved values to the data that remain.

For example, participants who stop treatment because of poor response may have missing efficacy outcomes that are systematically worse than observed outcomes. Similarly, severe adverse events may be under-recorded if participants with symptoms are less likely to complete follow-up. Treating either set of missing values as if it had the observed distribution can distort a treatment comparison.

These are assumptions about the missingness process, not conclusions established just by inspecting a data set. MAR allows missingness to depend on observed outcomes or covariates. For instance, if participants with worse recorded symptom scores are more likely to miss the next visit, the missingness may be MAR if those recorded scores are included in the model. If missingness depends on the unrecorded next-visit score even after conditioning on the observed data, it is MNAR.

A treatment comparison should specify its target before imputing: commonly, the expected utility for each treatment in a defined population and the difference between those expected utilities. If utility \(U\) is determined by efficacy and safety, its expectation depends on their joint distribution. When a utility function is nonlinear or includes interactions, imputing a pre-computed utility alone may fail to preserve the relationships needed to calculate expected utility. A practical approach is to impute the missing outcome components, calculate utility in each completed data set, and then estimate the treatment contrast.

Multiple Imputation for Expected Utility

Multiple imputation (MI) represents uncertainty about missing outcomes by creating \(M\) completed data sets. Each is generated by drawing missing values from an imputation model conditional on the observed data and its estimated parameters. The same analysis is then performed in each completed data set. The resulting estimates vary both because the observed sample is finite and because the missing values are uncertain.

Under MAR, a suitable imputation model should include treatment, relevant baseline covariates, observed efficacy and safety outcomes, and predictors of missingness and of the outcomes. When efficacy and safety outcomes are jointly relevant to utility, the imputation model should preserve their dependence. Imputation and analysis should also target the same population and outcome definition. The MAR assumption alone does not guarantee that a particular imputation model is correctly specified.

For a scalar estimand, let \(Q_m\) be the estimate from imputed data set \(m\), and let \(U_m\) be its estimated variance. The pooled estimate is \(\bar{Q}\), the average of the \(M\) estimates. The within-imputation variance \(\bar{U}\) is the average of the \(U_m\), and \(B\) is the sample variance of the \(Q_m\). Rubin's total variance combines both sources:

The first variance term reflects sampling uncertainty within completed data sets. The second reflects how much the estimate changes across plausible completions, with the factor \(1+1/M\) accounting for the finite number of imputations. A confidence interval can use a \(t\) reference distribution with MI degrees of freedom; it is not generally correct to use only the standard error from one completed data set.

Outcome-Dependent Missingness and Sensitivity Analysis

When missingness is MNAR, the distribution of an unobserved outcome cannot generally be recovered from observed data alone. The data do not reveal whether people with missing outcomes would, on average, have done better or worse than otherwise similar people whose outcomes were recorded. An MNAR analysis therefore requires an explicit assumption about the missing outcomes, and its consequences should be explored rather than concealed inside a single imputation model.

One practical approach is a pattern-mixture sensitivity analysis. First specify a primary MAR imputation model. Then, for missing outcomes, shift or otherwise alter the imputed distribution to represent a stated departure from MAR. For example, the assumed response probability among participants with missing efficacy data might be lower than the MAR-predicted probability. The shift should be defined on a clinically interpretable scale, applied consistently with the outcome's range, and examined over a plausible set of values. A selection model is an alternative: it directly models the probability of observation as a function of outcome and other data. Its outcome-dependence parameter is likewise not identified from observed data alone and must be varied or informed externally.

Derivation, Diagnostics, and Alternatives

The logic for calculating expected utility in the preceding example follows from partitioning participants into observed and missing outcome groups. If a fraction \(r\) has observed efficacy with response probability \(p_{\mathrm{obs}}\), and the remaining fraction \(1-r\) has an assumed response probability \(p_{\mathrm{mis}}\), the overall response probability is

This follows because response and non-response probabilities within the two groups combine according to the groups' proportions. In particular, the probability of response is the sum of the probabilities of being in each group and responding in that group. Under the stated additive utility \(U=100E-30S\), linearity of expectation gives \(E[U]=100P(E=1)-30P(S=1)\). If utility includes an interaction between efficacy and safety, the joint distribution is also needed; separate marginal imputations may then give the wrong expected utility.

Diagnostics can show whether the analysis is unstable or its assumptions are poorly represented, but they cannot prove MAR. Examine the amount and patterns of missingness by treatment and visit; assess whether the imputation model includes useful observed predictors; and check that imputed values are plausible and respect the outcome scale. For MI, inspect model fitting and convergence, compare observed and imputed distributions where comparison is meaningful, and verify that estimates and standard errors are not dominated by a few imputations. Large changes across imputations indicate substantial missing-data uncertainty. Differences in observed covariates between those with and without outcomes can help identify predictors worth including, but similar observed data do not establish that missing outcomes follow the MAR distribution.

Complete-case analysis discards participants with any required missing outcomes. It is simple, but can be biased and inefficient; it is not automatically valid merely because the percentage missing is modest. Likelihood-based analyses can use incomplete records under MAR when the outcome model and likelihood are appropriate. Inverse-probability weighting instead reweights observed records by their estimated observation probabilities, and can be useful under its own correct-model and positivity conditions. MI is often convenient for utility functions based on multiple outcome components, but it too relies on a defensible imputation model. None of these approaches removes the need to examine MNAR sensitivity when outcome-dependent missingness is plausible.

A utility comparison with missing outcomes is only as credible as its target, imputation model, and missingness assumptions. Report the primary assumption, the analysis used to propagate uncertainty, and sensitivity scenarios in terms that make clear how unobserved outcomes are being treated. This allows readers to see whether the treatment ranking or the size of the utility difference is robust to plausible departures from MAR.