Tutorials › Adaptive Clinical Trial Design › Missing And Unevaluable Joint Outcomes

Phase 1/2 Designs With Efficacy and Toxicity · Tutorial 639 of 799

Missing And Unevaluable Joint Outcomes

Learn to use partial outcome information in joint dose models, distinguish ignorable from outcome-dependent missingness, and assess the sensitivity of dose conclusions to unevaluable patients.

Advanced 10 min read

What You'll Learn

  • Represent efficacy and toxicity jointly, including which components are observed for each patient.
  • Identify the limitations of complete-case estimates for joint outcomes.
  • Construct observed-data likelihood contributions for complete, partial, and wholly missing outcomes.
  • State the assumptions that allow a missingness process to be ignored in outcome-model estimation.
  • Use explicit sensitivity assumptions to examine how unevaluable outcomes could affect a dose conclusion.
  • Diagnose informative missingness and report its implications transparently.

Why Missing Components Matter

Joint dose models use efficacy and toxicity together because the clinical meaning of a dose depends on both. If one component is unobserved, the patient is not necessarily unusable: an observed efficacy result can still inform the efficacy part of the model, and an observed toxicity result can inform the toxicity part. But neither component alone reveals the patient's joint outcome. Treating every partially assessed patient as a complete case discards information; treating an unknown component as if it had a particular value invents information.

The central task is to distinguish what the data establish from what an assumption supplies. The methods below concern unavailable or unevaluable efficacy and toxicity assessments at a dose. They do not prescribe how to define a clinical outcome when an event changes the meaning of the treatment question.

Definition: At dose \(d\), let \(E\) indicate efficacy (\(E=1\) for efficacy and \(E=0\) otherwise), and let \(T\) indicate toxicity (\(T=1\) for toxicity and \(T=0\) otherwise). A joint outcome is one of the four categories \((E,T)=(1,0),(1,1),(0,0),(0,1)\). An outcome component is missing or unevaluable when the available assessment does not permit its category to be classified under the trial's prespecified rules. Missingness is recorded separately for each component; an unknown value is not itself an outcome category.

Write the dose-specific joint probabilities as \(q_{10}(d)=P(E=1,T=0\mid d)\), \(q_{11}(d)=P(E=1,T=1\mid d)\), \(q_{00}(d)=P(E=0,T=0\mid d)\), and \(q_{01}(d)=P(E=0,T=1\mid d)\). They are nonnegative and sum to one. A joint dose model specifies how these probabilities vary with dose, possibly through efficacy and toxicity probabilities and an association parameter. The missing-data calculations do not require a particular parameterization, but they do require a model for the joint categories if partial observations are to inform joint dose probabilities.

Complete Cases and Their Limits

A complete-case analysis uses only patients for whom both efficacy and toxicity are classified. It can estimate the distribution of joint categories among those complete cases. If completeness is unrelated to the joint outcomes at a given dose, this distribution may represent the dose group, although fewer observations mean less precision. If completeness depends on efficacy, toxicity, or both, the complete cases can have a different outcome distribution from all treated patients. In that case, the complete-case proportions may be biased for the intended dose-group probabilities.

Even when complete cases are representative, excluding partial records wastes information about marginal outcomes. For example, an efficacy result observed while toxicity is unevaluable still tells us whether efficacy occurred. It does not tell us whether the patient was in the \((1,0)\) or \((1,1)\) joint category. The distinction matters: a marginal efficacy rate cannot, by itself, determine the probability of efficacy without toxicity.

Worked Example: What Complete Cases Omit

Suppose 40 hypothetical patients receive a dose. Both components are classified for 24 patients: 10 have \((E,T)=(1,0)\), 4 have \((1,1)\), 8 have \((0,0)\), and 2 have \((0,1)\). For 8 more patients, only efficacy is classified: 6 have \(E=1\) and 2 have \(E=0\). For 5, only toxicity is classified: 1 has \(T=1\) and 4 have \(T=0\). Neither component is classified for the remaining 3.

Among complete cases, the efficacy proportion is \((10+4)/24=14/24=0.5833\), and the toxicity proportion is \((4+2)/24=6/24=0.2500\). These are proportions among the 24 complete cases, not automatically among all 40 patients.

Using every observed efficacy result gives \(14+6=20\) efficacies among \(24+8=32\) patients with efficacy classified, or \(20/32=0.6250\). Using every observed toxicity result gives \(6+1=7\) toxicities among \(24+5=29\) patients with toxicity classified, or \(7/29=0.2414\). Thus the partial records add information about each marginal outcome.

The complete cases have 10 patients in the joint category \((1,0)\), so the complete-case estimate of that category's fraction is \(10/24=0.4167\). The partial records do not reveal how many additional patients were in \((1,0)\): the six efficacy-positive patients with unknown toxicity might have toxicity or not. Consequently, the joint category probability cannot be recovered from the observed marginal counts alone without further modeling or assumptions.

Observed-Data Likelihoods

A likelihood-based analysis can retain the information in every observed component by summing over the unobserved component. For clarity, condition on the assigned dose and the trial information used to assign it. For one patient, the outcome-model likelihood contribution is the probability of the observed data under the joint model. A fully observed patient contributes the probability of the observed joint category. An efficacy-only patient contributes the sum of probabilities for the two joint categories consistent with that efficacy result. A toxicity-only patient contributes the corresponding sum over efficacy. If both components are missing, the outcome contribution is the sum over all categories, which is one.

Observed-data likelihood contribution: At dose \(d\), a fully observed \((e,t)\) contributes \(q_{et}(d)\). An efficacy-only observation \(E=e\) contributes \(\sum_{t=0}^1q_{et}(d)\). A toxicity-only observation \(T=t\) contributes \(\sum_{e=0}^1q_{et}(d)\). A wholly unobserved outcome contributes \(\sum_{e=0}^1\sum_{t=0}^1q_{et}(d)=1\) to the outcome likelihood.

The likelihood across patients is the product of their contributions, or equivalently the sum of their log contributions on the log scale. For a model with parameters shared across doses, each contribution uses the probabilities at that patient's assigned dose. If a patient has a known efficacy value but unknown toxicity, the contribution is a marginal efficacy probability, not a guessed joint category.

Worked Example: Likelihood Contributions From Partial Outcomes

At one hypothetical dose, suppose a fitted joint model assigns probabilities \(q_{10}=0.40\), \(q_{11}=0.20\), \(q_{00}=0.25\), and \(q_{01}=0.15\). These sum to \(1.00\). Consider five records: one complete \((1,0)\), one complete \((0,1)\), one with efficacy observed as \(E=1\) only, one with toxicity observed as \(T=0\) only, and one with neither component observed.

The two complete records contribute \(q_{10}=0.40\) and \(q_{01}=0.15\). The efficacy-only positive record contributes \(q_{10}+q_{11}=0.40+0.20=0.60\), because either toxicity value is compatible with the observation. The toxicity-only negative record contributes \(q_{10}+q_{00}=0.40+0.25=0.65\). The wholly missing record contributes \(1\).

Ignoring factors associated only with the recorded missingness patterns, the outcome likelihood contribution for these five records is \[ 0.40\times0.15\times0.60\times0.65\times1=0.02340. \] The partial observations inform the joint model through its marginal probabilities. Assigning the efficacy-only patient to \((1,0)\), for instance, would incorrectly use \(0.40\) instead of the appropriate \(0.60\).

When Can the Missingness Process Be Ignored?

The observed-data likelihood above can be used without a separate model for missingness under an ignorability assumption. In a common formulation, missingness is missing at random when, conditional on the components actually observed and the relevant recorded information, the probability of the observation pattern does not depend on the unobserved outcome components. For example, among patients with the same observed efficacy and recorded history at a dose, the chance that toxicity is unclassified must not additionally depend on the patient's unobserved toxicity value.

For likelihood inference, ignorability also requires that the parameters governing the outcome distribution and the missingness process be distinct. Under these conditions, the missingness likelihood can be factored away for inference about the outcome-model parameters. These conditions do not mean that missingness is unrelated to all recorded information; it may depend on dose, observed components, or recorded history, provided that the dependence is properly represented in the conditioning information.

A short derivation shows why summation is the correct likelihood operation. Let \(Y=(E,T)\), let \(R\) record which components are observed, and let \(Y_{\mathrm{obs}}\) and \(Y_{\mathrm{mis}}\) be the observed and missing components. The joint probability of the observed data and the observation pattern can be written as

$$ P(Y_{\mathrm{obs}},R\mid d) = \sum_{Y_{\mathrm{mis}}} P(R\mid Y_{\mathrm{obs}},Y_{\mathrm{mis}},d) P(Y_{\mathrm{obs}},Y_{\mathrm{mis}}\mid d). $$

Under missing at random, \(P(R\mid Y_{\mathrm{obs}},Y_{\mathrm{mis}},d)=P(R\mid Y_{\mathrm{obs}},d)\), so this factor does not vary across the possible missing outcome values. It can be taken outside the sum:

$$ P(Y_{\mathrm{obs}},R\mid d) = P(R\mid Y_{\mathrm{obs}},d) \sum_{Y_{\mathrm{mis}}}P(Y_{\mathrm{obs}},Y_{\mathrm{mis}}\mid d). $$

The sum is precisely the marginal probability of the observed components under the joint outcome model. The first factor concerns the observation pattern, not the outcome-model parameters under parameter distinctness, so it can be omitted from likelihood inference about those parameters. If the missingness probability depends on an unobserved outcome, this factor cannot generally be taken outside the sum; an outcome-only likelihood is then insufficient to identify the relevant quantities.

Sensitivity Assumptions for Informative Missingness

When missingness may depend on an unobserved component, the data alone generally cannot determine how those missing outcomes are distributed. A sensitivity analysis makes the uncertainty explicit by varying a parameter or assumption about the missing outcomes, then recomputing dose-specific estimates or decision summaries. Options include specifying category probabilities among patients with a particular missingness pattern, or applying a shift to the odds of an unobserved outcome relative to comparable observed patients. The chosen assumptions should be clinically plausible and reported as assumptions, not as facts learned from the missing records.

Worked Example: Sensitivity of an Acceptable-Outcome Rate

At a hypothetical dose, 20 patients are treated. Both components are classified for 12: 5 have the acceptable joint outcome \((E,T)=(1,0)\), 2 have \((1,1)\), 4 have \((0,0)\), and 1 has \((0,1)\). Both components are unclassified for the other 8 patients. Suppose the decision rule under consideration requires the acceptable-outcome probability to exceed 0.45. This calculation is a simple sensitivity illustration, not a complete inferential dose-selection procedure.

Let \(s\) denote the assumed fraction of the 8 unevaluable patients who would have had the acceptable outcome. The completed-data fraction under that assumption is \[ \frac{5+8s}{20}. \] If \(s=0.25\), the fraction is \((5+8(0.25))/20=7/20=0.3500\), below the illustrative threshold. If \(s=0.75\), it is \((5+8(0.75))/20=11/20=0.5500\), above the threshold.

The observed complete cases are unchanged in both calculations; only the assumption about the missing joint outcomes differs. Since the conclusion crosses the threshold, the dose judgment is sensitive to this assumption. A model-based analysis would propagate uncertainty in the outcome probabilities as well, but it would still need a defensible missingness assumption or an explicit sensitivity model.

Diagnostics, Comparisons, and Limitations

Before selecting an analysis, summarize assessment patterns by dose and by relevant recorded characteristics: both components observed, efficacy only, toxicity only, or neither. Report the number of patients in each pattern and compare the observed characteristics and outcomes across patterns. Such comparisons can reveal warning signs—for example, an unusually high rate of unevaluable toxicity among patients with severe recorded symptoms—but they cannot prove that missingness is ignorable. Dependence on an unobserved outcome is not directly testable from the observed data alone.

The main approaches answer different questions and rely on different assumptions:

ApproachWhat it usesMain concern
Complete-case analysisPatients with both components classifiedCan lose precision and can be biased if complete cases are not representative
Observed-data likelihoodAll classified components, marginalizing over missing onesRequires a suitable joint outcome model and an ignorable missingness assumption, unless missingness is modeled
Sensitivity analysisObserved data plus explicit assumptions about unobserved outcomesResults depend on assumptions that the observed data may not identify

Common failure modes include coding every unevaluable efficacy result as no efficacy, coding every unobserved toxicity result as no toxicity, and silently treating incomplete patients as representative complete cases. A conservative rule that classifies some missing outcomes adversely can be a prespecified operational choice, but it is not a neutral statistical consequence of missingness. It should be distinguished from likelihood-based inference and justified clinically.

Another limitation is that no method can recover information that was never observed without assumptions. A wholly missing joint outcome contributes no outcome information to the likelihood when its missingness is ignorable. Partial outcomes do contribute, but cannot by themselves reveal the dependence between efficacy and toxicity. Model uncertainty, small dose-specific samples, and sparse joint categories can further make the resulting dose probabilities unstable. Accordingly, report missingness patterns, the primary assumption, and whether plausible sensitivity scenarios alter the dose conclusion.

Key takeaway: Keep the observed components, missingness pattern, and assumed unobserved outcomes distinct. Use partial observations through marginal likelihood contributions when the assumptions support it, and show how conclusions change when informative missingness is plausible.

Check Your Understanding

Answer each question, then compare with the solutions below.

  1. At a dose, an efficacy-only record has \(E=0\). Which joint probabilities contribute to its outcome likelihood?
  2. A toxicity-only record has \(T=1\). If \(q_{11}=0.18\) and \(q_{01}=0.12\), calculate its likelihood contribution.
  3. In the first worked example, why is \(10/24\) not automatically an estimate of the acceptable joint-outcome fraction among all 40 treated patients?
  4. Under what missingness condition can the observation-pattern likelihood be ignored for outcome-model inference?
  5. In the sensitivity example, what value of \(s\) makes the completed-data fraction exactly 0.45?

Solutions. (1) The record is consistent with \((E,T)=(0,0)\) or \((0,1)\), so the contribution is \(q_{00}+q_{01}\). (2) Either efficacy value is compatible with observed toxicity, so the contribution is \(q_{11}+q_{01}=0.18+0.12=0.30\). (3) The 10 observed acceptable outcomes are among complete cases only; the 16 patients without both components may include additional acceptable outcomes, and their joint categories are not known. (4) Missing at random, conditional on the observed components and relevant recorded information, together with distinct outcome and missingness parameters, permits the missingness factor to be ignored for likelihood inference about outcome parameters. (5) Solve \((5+8s)/20=0.45\): \(5+8s=9\), so \(s=4/8=0.50\).

Clinical Biostats