Why Equally Good Fits Can Lead to Different Conclusions
In a randomized trial, a model can describe the observed outcomes and follow-up patterns well while still giving a highly assumption-dependent answer about outcomes that were not observed. This is especially important under missing not at random (MNAR): after conditioning on the information in the model, the probability that an outcome is missing may still depend on its unobserved value. That dependence cannot generally be estimated from the observed outcomes alone.
The comparison here uses two standard ways to express assumptions about those unseen outcomes. A pattern-mixture model specifies the outcome distribution within each missingness pattern. A selection model specifies the outcome distribution and models the probability of observation conditional on the outcome. Neither label, by itself, determines the answer. The treatment conclusions differ when their particular assumptions about missing outcomes differ.
Consider a hypothetical randomized trial with a binary favorable outcome at follow-up. Let \(A=1\) denote treatment and \(A=0\) control; let \(Y=1\) mean a favorable outcome and \(Y=0\) an unfavorable outcome; and let \(R=1\) mean that the follow-up outcome was observed. Assume the outcome is conceptually defined for every randomized participant at the target time. The analysis below estimates the risk in each randomized arm, including participants with \(R=0\); it is not a comparison restricted to observed participants.
Hypothetical Trial Data and a Pattern-Mixture Analysis
Suppose each arm contains 200 participants. The observed results are:
| Arm | Favorable, observed | Unfavorable, observed | Outcome missing | Total |
|---|---|---|---|---|
| Treatment | 105 | 45 | 50 | 200 |
| Control | 96 | 64 | 40 | 200 |
Among participants with observed outcomes, the favorable-outcome risks are \(105/150=0.7000\) in the treatment arm and \(96/160=0.6000\) in the control arm. These are observed-pattern summaries, not yet risks for all randomized participants. A pattern-mixture analysis needs an assumption about the favorable-outcome risk among the participants whose outcomes are missing.
Worked Example: Pattern-Mixture Risks Under a Common Odds Shift
Assume \(\delta=-\log(2)\), so the odds of a favorable outcome among participants with missing outcomes are half the odds among observed participants in the same arm. Calculate the all-randomized risks and their treatment-minus-control difference.
For treatment, \(q_1=105/150=0.7\), with observed odds \(0.7/0.3=7/3\). Halving these odds gives missing-outcome odds \(7/6\), so \(m_1=(7/6)/(1+7/6)=7/13\), or approximately 0.5385. Thus the treatment risk is
For control, \(q_0=96/160=0.6\), with observed odds \(0.6/0.4=3/2\). Halving these odds gives \(3/4\), so \(m_0=(3/4)/(1+3/4)=3/7\), or approximately 0.4286. The control risk is
The estimated treatment-minus-control risk difference is \(0.6596-0.5657=0.0939\), or about 9.39 percentage points. Under this stated assumption, the estimated favorable-outcome risk is higher with treatment. This is a model-based point estimate, not by itself a test of statistical significance or evidence that the assumed missing-outcome distributions are correct.
A Selection Model for the Same Observed Patterns
A selection model describes how observation depends on the outcome. For arm \(a\), write \(s_{1a}=P(R=1\mid Y=1,A=a)\) and \(s_{0a}=P(R=1\mid Y=0,A=a)\). One useful sensitivity parameter is the response odds ratio
Thus \(\gamma_a>1\) means that participants with favorable outcomes have higher odds of being observed than participants with unfavorable outcomes, within arm \(a\). This is an assumption about selection, not an estimate that the observed trial outcomes identify under MNAR.
The connection to the pattern-mixture quantities is exact for this binary outcome. Among observed participants, the outcome odds are the population outcome odds multiplied by \(s_{1a}/s_{0a}\). Among missing participants, they are the population outcome odds multiplied by \((1-s_{1a})/(1-s_{0a})\). Taking the ratio of these two sets of odds gives:
Consequently, a large \(\gamma_a\) implies lower favorable-outcome odds in the missing pattern than in the observed pattern. For a chosen positive \(\gamma_a\), the observed response rate and observed outcome proportion can be retained while the implied missing-outcome risk changes. The parameter changes the extrapolation, not the observed counts.
Worked Example: A Selection Assumption Reverses the Contrast
Keep the hypothetical trial data above. For control, assume \(\gamma_0=2\); for treatment, assume \(\gamma_1=20\). Calculate the implied missing-outcome risks and the all-randomized treatment contrast.
For control, the observed odds are \(96/64=3/2\). Dividing by \(\gamma_0=2\) gives missing-outcome odds \(3/4\), so \(m_0=3/7\). Using the unrounded fraction throughout, the control risk is
For treatment, the observed odds are \(105/45=7/3\). Dividing by \(\gamma_1=20\) gives missing-outcome odds \(7/60\), so \(m_1=7/67\). Therefore,
The selection-model treatment-minus-control risk difference is \(0.5511-0.5657=-0.0146\), or about −1.46 percentage points. Under this selection assumption, the estimated risk is slightly lower with treatment. This differs in direction from the pattern-mixture estimate of +9.39 percentage points. The contrast in conclusions arises from different assumptions about missing outcomes, not from a different set of observed trial data.
Why Both Models Can Fit the Observed Data
The observed data reveal the number of participants with \(R=1\) in each arm and, among them, the numbers with \(Y=1\) and \(Y=0\). They do not reveal how many of the participants with \(R=0\) would have had \(Y=1\). In a pattern-mixture analysis, that unknown is represented through \(m_a\) or an equivalent sensitivity parameter. In a selection analysis, it is represented through assumptions such as \(\gamma_a\), together with the outcome and response model.
For this binary example, every chosen \(m_a\) between 0 and 1 gives a corresponding population risk \(p_a=P(Y=1\mid A=a)\), since \(p_a=P(R=1\mid A=a)q_a+P(R=0\mid A=a)m_a\). A compatible selection mechanism can then be constructed. Conversely, a specified selection odds ratio implies an \(m_a\) through the odds relationship above. Thus the parameterizations can describe the same observed patterns while implying different unseen outcomes.
This distinction matters: it is not true that selection models and pattern-mixture models must disagree merely because they have different names. With equivalent assumptions and parameterizations, their implied distributions can agree. They disagree in the case study because the selected sensitivity assumptions imply different missing-outcome distributions. Similar—or identical—fit to observed data does not settle which extrapolation is more credible.
Worked Example: The Selection Value at Which the Contrast Changes Sign
Keep the control assumption \(\gamma_0=2\), which implies \(m_0=3/7\). Find the treatment selection odds ratio \(\gamma_1\) at which the treatment-minus-control risk difference is zero.
For the two risks to be equal, their numerators over the common denominator 200 must be equal. Therefore the treatment missing-outcome risk \(m_1\) must satisfy
The odds corresponding to \(m_1=57/350\) are \(57/293\). Treatment observed-outcome odds are \(105/45=7/3\). Since the observed odds divided by the missing odds equal \(\gamma_1\),
With \(\gamma_0=2\), the estimated contrast is positive when \(\gamma_1\) is below approximately 11.99, zero at that value, and negative when it is above it. The threshold describes sensitivity to the assumed selection relationship. It does not estimate the true \(\gamma_1\) from the trial.
Diagnostics, Alternatives, and Limitations
Diagnostics can reveal problems with the observed-data model, but cannot generally verify an MNAR extrapolation from observed data alone. A careful analysis should check that the outcome and response models reproduce the observed counts by randomized arm and relevant observed covariates; examine sparse cells and unstable estimates; and report response rates and observed outcome distributions by arm. If the model fails to reproduce those observable features, its extrapolation is not a remedy for the poor fit.
If two MNAR models reproduce the observed data equally well, ordinary goodness-of-fit comparisons based only on those data cannot establish which missing-outcome assumption is right. Useful checks instead include examining whether the sensitivity parameters correspond to defensible clinical scenarios, varying them across a prespecified range, and showing how the treatment contrast changes. The earlier material on delta sensitivity and tipping points provides complementary ways to organize such sensitivity analyses.
| Approach | What it specifies | Key limitation for this comparison |
|---|---|---|
| Pattern-mixture | Outcome distribution within observed and missing patterns | The missing-pattern distribution is not determined by observed outcomes. |
| Selection | Observation probability conditional on the outcome and other modeled variables | The outcome-dependent selection parameter is not generally identified under MNAR. |
| MAR analysis | Missingness independent of the unobserved outcome, given included observed information | It is a different, untestable assumption when missingness may depend on the unseen outcome. |
The MAR analysis is a useful comparator, but it does not resolve the MNAR question: it answers under a different assumption. Likewise, a model-based point estimate does not by itself account for sampling uncertainty or uncertainty about the sensitivity parameter. If intervals or tests are reported, their construction should reflect the fitted model and its uncertainty; they do not validate the MNAR assumption.
The numerical illustration also relies on a binary outcome defined for every randomized participant at the target time. If an outcome is undefined after an event such as death, treating that value as merely missing would change the question and require a clearly specified estimand. In addition, the example uses arm-level risks without covariate adjustment; real analyses may need to account for baseline variables, repeated measurements, and the trial’s randomization and analysis plan.
A Common Mistake
When summarizing such an analysis, distinguish three things: what was observed, what was assumed about missing outcomes, and what estimate followed from combining them. For the hypothetical trial, the observed counts are fixed; the estimated risk difference changes from +9.39 percentage points under the common pattern-mixture odds shift to −1.46 percentage points under the stated selection odds ratios. These are conditional, assumption-dependent conclusions—not competing summaries of different observed datasets.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In the hypothetical trial, what does \(\gamma_a=1\) mean for the response mechanism in arm \(a\)?
- If \(q_a=0.6\) and \(\gamma_a=3\), calculate the missing-pattern favorable-outcome risk \(m_a\).
- Why can equal observed-data fit fail to choose between two MNAR analyses?
- In the worked tipping-point calculation, what happens to the treatment contrast as \(\gamma_1\) increases beyond approximately 11.99, with \(\gamma_0=2\) held fixed?
Solutions. (1) The odds of being observed are equal for \(Y=1\) and \(Y=0\), conditional on that arm, because their ratio is \(\gamma_a=1\). (2) The observed odds are \(0.6/0.4=1.5\); dividing by 3 gives missing-pattern odds 0.5, so \(m_a=0.5/(1+0.5)=1/3\). (3) The observed data contain no outcomes for participants with \(R=0\), so distinct MNAR assumptions can imply different unseen outcomes without changing the fit to observed outcomes. (4) The implied treatment missing-outcome risk falls, and the treatment-minus-control contrast becomes negative.