Observed Fit and Unobserved Outcomes
When participants leave a longitudinal study, later outcomes are not recorded for them. Two models for dropout can therefore agree on everything visible in the data and still make quite different claims about the outcomes that would have been measured after dropout. This distinction is central to comparing missing-not-at-random (MNAR) models: a close match to observed data is not, by itself, evidence that two models extrapolate in the same way.
Consider a study with an outcome \(Y_1\) measured at an initial visit and a later outcome \(Y_2\). Let \(X\) denote the observed information available before the later outcome, including baseline covariates and \(Y_1\). Define \(R=1\) if \(Y_2\) is observed and \(R=0\) if the participant drops out before it is measured. For a participant with \(R=0\), \(X\) is observed but \(Y_2\) is not.
The examples below hold fixed the distribution of \(X\), the response probability given \(X\), and the distribution of \(Y_2\) among participants who remain observed. They vary only the distribution assigned to \(Y_2\) among dropouts. Thus the models have the same observed-data fit by construction, while their extrapolations can be compared directly.
A Pair of Compatible MNAR Models
For a transparent numerical illustration, suppose we focus on one value of \(X\), and let \(m=m(X)\) be the mean later outcome among participants with \(R=1\). Use the following common observed-data components in both candidate models:
The second argument of \(N(m,4)\) is the variance, so the standard deviation is \(2\). Candidate \(d\) assigns dropouts the distribution
where \(d>0\) is the assumed difference, in outcome units, between the mean among observed participants and the mean among dropouts. We compare \(d=2\) and \(d=6\). Each candidate is MNAR: the response probability depends on the unobserved \(Y_2\), as shown by the selection model implied by these distributions.
These assumptions specify a complete-data distribution by mixing the respondent and dropout distributions with their pattern probabilities:
Bayes' rule then gives the response probability conditional on a particular outcome value. This step matters: the specified pattern-specific outcome distributions imply a particular selection mechanism; the response probability is not chosen independently of them.
The corresponding response odds are \(qg(y\mid X)/\{(1-q)h_d(y\mid X)\}\). For \(q=0.75\), the factor \(q/(1-q)\) is \(3\). Since both normal densities have variance \(4\), their log density ratio is
In particular, the last term is positive: writing \(z=y-m\), the difference of squared deviations is \((z+d)^2-z^2=2dz+d^2\). Therefore the implied selection model is
Worked Example: Same Observed Likelihood, Different Dropout Assumptions
Suppose a hypothetical longitudinal study has 200 participants with the same \(X\). Of these, 150 have \(Y_2\) observed and 50 drop out before that visit. Suppose both candidate models use \(q=0.75\) and the same respondent density \(g=N(12,4)\); they differ only in whether dropout outcomes follow \(N(10,4)\) or \(N(6,4)\).
Under either candidate, the response-pattern part of the observed-data log likelihood is
The outcome part of the observed-data log likelihood is the sum of \(\log g(y_i\mid X)\) over the 150 observed outcomes. That term is identical under both models because both use \(g=N(12,4)\). No \(Y_2\) values are recorded for the 50 dropouts, so their unobserved values do not enter the observed-data likelihood. Thus the two candidates have the same observed-data likelihood for these data, even though one assigns dropouts a mean of 10 and the other a mean of 6.
This is a statement about the information the data provide, not a claim that the two models describe the same full population. In particular, it does not say that their implied distributions after dropout agree.
What Selection Probabilities Do the Models Imply?
The induced selection probabilities help make the difference between the candidates concrete. They vary with \(y\), because the probability of remaining observed depends on the outcome that may be missing. The following calculations use \(m=12\). First evaluate the response probability at \(y=m\).
Worked Example: Correct Selection Probabilities at Two Outcome Values
For Candidate A, \(d=2\). At \(y=m=12\), its logit is \[ \eta_A=\log 3+\frac{2(12-12)}{4}+\frac{2^2}{8} =\log 3+0.5 \approx1.5986. \] Thus \[ P(R=1\mid Y_2=12,X) = \frac{\exp(1.5986)}{1+\exp(1.5986)} \approx0.8318. \]
For Candidate B, \(d=6\). At the same outcome value, \[ \eta_B=\log 3+\frac{6(12-12)}{4}+\frac{6^2}{8} =\log 3+4.5 \approx5.5986, \] so \[ P(R=1\mid Y_2=12,X) = \frac{\exp(5.5986)}{1+\exp(5.5986)} \approx0.9963. \]
At the candidate-specific dropout mean \(y=m-d\), the calculations differ. For Candidate A, \(y=10\), so \[ \eta_A=\log 3+\frac{2(10-12)}{4}+\frac{2^2}{8} =\log 3-0.5\approx0.5986, \] giving a response probability of approximately \(0.6453\). For Candidate B, \(y=6\), so \[ \eta_B=\log 3+\frac{6(6-12)}{4}+\frac{6^2}{8} =\log 3-4.5\approx-3.4014, \] giving a response probability of approximately \(0.0322\). These are probabilities at particular outcome values, not the overall response rate. Under either candidate, averaging the selection probability over its full outcome distribution gives \(P(R=1\mid X)=0.75\).
That last equality can be checked directly, without relying on a numerical approximation. Multiply the selection probability by the marginal outcome density and simplify:
Integrating both sides over \(y\) gives \(P(R=1\mid X)=q\), because \(g\) is a density and integrates to one. The selection probabilities at individual values therefore vary sharply between candidates while their marginal response rates remain the same.
Different Extrapolations for the Full Population
A practical consequence of different dropout distributions is that a full-population summary can differ even when the observed-data distributions match. For the normal candidates above, the expected later outcome in the full population is the pattern-weighted mean:
Worked Example: Comparing Full-Population Means
Keep \(m=12\) outcome points and \(q=0.75\). Under Candidate A, \(d=2\), so its dropout mean is \(12-2=10\), and its full-population mean is \[ 0.75(12)+0.25(10)=9+2.5=11.5\text{ points}. \] Under Candidate B, \(d=6\), so its dropout mean is \(12-6=6\), and its full-population mean is \[ 0.75(12)+0.25(6)=9+1.5=10.5\text{ points}. \] The two candidates differ by \(11.5-10.5=1.0\) point in the full-population mean, despite having the same observed response rate and the same respondent outcome distribution. This difference is due entirely to the assumed outcome distribution after dropout.
For these normal mixtures, the full-population variance also differs. The common within-pattern variance is \(4\), and the between-pattern contribution is \(q(1-q)d^2\). Candidate A therefore has variance \[ 4+(0.75)(0.25)(2^2)=4+0.75=4.75\text{ points}^2, \] whereas Candidate B has variance \[ 4+(0.75)(0.25)(6^2)=4+6.75=10.75\text{ points}^2. \] This additional calculation reinforces that agreement about observed outcomes does not fix the full-population distribution.
How to Assess Agreement and Where It Fails
Observed-data checks can assess whether the candidates represent the recorded information adequately. For example, one can compare their fitted response rates across values of \(X\), their distributions of \(Y_2\) among respondents, and their fit to other recorded variables. If either candidate fails to reproduce these features, it is not a satisfactory description of the observed data. But if both reproduce them, the observed-data fit alone cannot select between their different assumptions about dropout outcomes.
A useful diagnostic is to check the internal consistency of a candidate. Its implied selection probabilities must lie between zero and one, and averaging them over its implied outcome distribution must recover the stated response rate. The Bayes construction above ensures both properties for the specified densities. In practical models, numerical integration or simulation can check the marginal response rates; a mismatch may indicate an implementation error or incompatible model components.
Other warning signs concern model use rather than observed-data fit. A candidate may imply near-zero or near-one response probabilities over much of the outcome range, making its assumptions consequential and sensitive to small changes. A normal distribution may also be unsuitable for a bounded or highly skewed outcome. Such features call for scrutiny and sensitivity analysis, but they do not reveal the true missing outcomes from the observed data alone.
The key alternative is not simply a different fitting criterion. A model that assumes response is independent of the unobserved outcome given recorded history imposes a different missingness assumption; a pattern-mixture specification directly describes outcome distributions by observation pattern. Either framework still needs assumptions about what is unobserved. Comparing observed-data fit is useful for detecting poor representation of recorded information, but it cannot resolve differences in extrapolation that leave that information unchanged.
This limitation is an identification issue. The observed data reveal the response rate and the distribution of outcomes among respondents, but not the distribution of outcomes among dropouts. Without additional information or restrictions, multiple full-data distributions can be compatible with the same observed-data distribution. Consequently, selecting among compatible MNAR candidates requires substantive justification, external information, or sensitivity analysis—not merely a preference for one candidate's observed-data fit.
Check Your Understanding
Use the model definitions in this tutorial. The solutions follow the questions.
- If \(q=0.75\), \(m=12\), and \(d=4\), what mean does the candidate assign to \(Y_2\) among dropouts?
- For \(q=0.75\), what is the response odds multiplier \(q/(1-q)\) in the implied selection model?
- Why do candidates with the same \(q\) and \(g\) have the same observed-data likelihood, even if their \(h_d\) distributions differ?
- For \(m=12\), \(q=0.75\), and \(d=4\), calculate the full-population mean and state its units if \(Y_2\) is measured in points.
Solutions. (1) The dropout mean is \(m-d=12-4=8\) points. (2) The multiplier is \(q/(1-q)=0.75/0.25=3\). (3) The observed likelihood contains the response-pattern probabilities and the outcome densities for observed \(Y_2\) values. Those terms are identical when \(q\) and \(g\) are shared; the \(h_d\) values describe outcomes that are not recorded for dropouts. (4) The full-population mean is \(qm+(1-q)(m-d)=0.75(12)+0.25(8)=9+2=11\) points.