What the Models Say About Outcomes After Dropout
When a participant drops out, later outcomes are not observed, but a fitted missing-not-at-random (MNAR) model can still imply a distribution for them. Comparing those distributions makes a model's extrapolation easier to inspect: two models can fit the observed data similarly yet imply different typical values, variability, or tails after dropout. This comparison complements the earlier material on fit agreement versus extrapolation agreement; its focus is the distributions themselves, not a single fit score.
A distributional comparison is meaningful only after stating the outcome and target group. For example, the target could be the week-12 symptom score among participants who dropped out after week 8, conditional on their observed week-8 score and other observed history. The model-implied distribution answers a conditional prediction question for that group. It does not reveal what those unobserved scores actually were.
For a longitudinal study, \(Y_{\text{mis}}\) may be a single future outcome or a vector of outcomes after dropout. The same principle applies to a dropout pattern: condition on the observed history and the pattern, then summarize the distribution of the unobserved outcomes under each candidate model. Be explicit about whether the comparison is conditional on a particular history, averaged over observed dropout participants, or reported separately for different patterns.
Why a Selection Model Implies a Missing-Outcome Distribution
A selection model specifies an outcome distribution together with a model for dropout that can depend on the outcome. To see how the dropout mechanism changes the distribution, consider one future outcome \(Y\), observed history \(H\), and dropout indicator \(D\). If \(D=1\) means that \(Y\) is missing because of dropout, Bayes' rule gives the model-implied distribution among dropouts.
The numerator weights each possible outcome by its probability under the outcome model and its probability of dropout at that outcome. The denominator is the overall dropout probability conditional on \(H\); integrating the numerator over all possible \(y\) gives that probability, so the displayed density integrates to one. If larger outcomes make dropout more likely, the weighting can shift the dropout group's distribution toward larger values. The direction and size of the shift depend on the specified models.
This is a model-based implication, not a distribution directly observed among dropouts. After dropout, the data do not reveal \(Y\) for those participants. Consequently, summaries can be calculated from fitted model predictions, but agreement with the observed outcomes cannot by itself verify the model's predictions for the unobserved outcomes.
Comparing the Distributional Summaries
For each candidate, calculate summaries on a common outcome scale and for a clearly specified target group. The mean describes the distribution's center, the variance describes its spread around the mean, and quantiles describe locations in its cumulative distribution. A useful comparison often reports several quantiles, such as the 10th, 50th, and 90th percentiles, rather than relying on a single center.
For a target group with different observed histories, one can report conditional predictions for individual histories or average predictions over the same set of dropout participants. If the goal is the average predicted mean, for example, average each candidate's conditional mean across those participants using the same weights. Do not confuse that average of conditional means with the variance or quantiles of the pooled predictive distribution: pooling also reflects differences among participants' predicted means.
| Summary | What it describes | What it may conceal |
|---|---|---|
| Mean | Average implied outcome in the target group | Skewness, tail behavior, or distinct subgroups |
| Variance | Spread around the implied mean | Where outcomes fall in the distribution or whether tails differ |
| Quantiles | Outcome values below which specified proportions of the distribution lie | Differences between reported quantiles or features between them |
| Density or distribution plot | Shape, including multiple modes and tail differences | It can be difficult to read small differences precisely |
Worked Example: Comparing Means, Variances, and Quantiles
Suppose two hypothetical candidate models imply distributions for a post-dropout symptom score, measured in points, in the same dropout group. Candidate A gives \(Y_{\text{mis}}\sim N(50,100)\), and candidate B gives \(Y_{\text{mis}}\sim N(45,144)\). The second argument is the variance, so the standard deviations are 10 and 12 points, respectively. Assume these are the fitted predictive distributions for the same target group, on the same score scale.
For a normal distribution, the \(q\)-quantile is \(\mu+z_q\sigma\), where \(z_q\) is the standard-normal \(q\)-quantile. The means are 50 points under A and 45 points under B; the variances are 100 and 144 points squared. Thus B implies a lower center and greater spread.
For the 10th percentile, use \(z_{0.10}\approx-1.2816\). Under A, the calculation is \(50+(-1.2816)(10)=37.184\), or about 37.2 points. Under B, it is \(45+(-1.2816)(12)=29.6208\), or about 29.6 points. The 10th percentile is lower under B.
For the 90th percentile, \(z_{0.90}\approx1.2816\). Under A, \(50+(1.2816)(10)=62.816\), or about 62.8 points. Under B, \(45+(1.2816)(12)=60.3792\), or about 60.4 points. In this comparison, B has a lower 10th percentile and a lower 90th percentile despite having a larger variance. That is possible because B's mean is also lower; variance alone does not determine the location of a quantile.
These are hypothetical model implications, not observed scores among participants after dropout. The comparison describes a distributional difference in predicted symptom points; it does not establish which candidate's missing-outcome distribution is correct.
Pooling Across Dropout Histories
A single summary for all dropouts can hide differences across histories or dropout patterns. If model predictions vary by participant, the distribution pooled over those participants includes both within-participant predictive uncertainty and between-participant differences in predicted means. State the target weights used for pooling. In particular, use the same dropout participants and weights for each candidate if the purpose is to compare their predictions for the same target group.
Worked Example: Why a Pooled Variance Can Differ
Consider two hypothetical observed-history groups among dropouts. Group 1 has weight 0.75 and group 2 has weight 0.25. Under candidate A, their post-dropout outcome distributions, measured in score points, are \(N(10,4)\) and \(N(18,9)\). Under candidate B, they are \(N(11,4)\) and \(N(16,9)\). These are conditional distributions, and the same group weights apply to both candidates.
The pooled mean under A is \(0.75(10)+0.25(18)=12\) points. Under B it is \(0.75(11)+0.25(16)=12.25\) points. The pooled centers are close, but that alone does not establish similar distributions.
For a mixture with group means \(\mu_j\), within-group variances \(\sigma_j^2\), and weights \(w_j\), the pooled variance is \(\sum_j w_j\{\sigma_j^2+(\mu_j-\mu_{\text{pool}})^2\}\). This follows by decomposing squared deviation from the pooled mean into deviation from the group mean plus deviation of the group mean from the pooled mean; the cross-product averages to zero within each group.
For A, the pooled variance is \(0.75\{4+(10-12)^2\}+0.25\{9+(18-12)^2\}\), which equals \(0.75(8)+0.25(45)=17.25\) points squared. For B it is \(0.75\{4+(11-12.25)^2\}+0.25\{9+(16-12.25)^2\}\), which equals \(0.75(5.5625)+0.25(23.0625)=9.9375\) points squared. Thus similar pooled means coexist with substantially different pooled variances. The difference reflects both conditional variances and separation between the group means.
A Discrete Example of Selection Weighting
The Bayes formula also shows concretely how different selection assumptions can produce different missing-outcome distributions. The following hypothetical calculation uses only two possible outcome values so that every probability and summary can be checked directly.
Worked Example: Dropout Probabilities Reweight Possible Outcomes
Suppose a hypothetical outcome \(Y\), measured in points, can be 40 or 60, each with probability 0.5 conditional on a participant's observed history. Under candidate A, the probability of dropout is 0.2 when \(Y=40\) and 0.8 when \(Y=60\). The overall dropout probability is \(0.5(0.2)+0.5(0.8)=0.5\).
Among dropouts, the conditional probability of \(Y=60\) is \(P(Y=60\mid D=1)=P(D=1\mid Y=60)P(Y=60)/P(D=1)=(0.8)(0.5)/0.5=0.8\). Therefore the implied distribution puts probability 0.2 on 40 and 0.8 on 60. Its mean is \(0.2(40)+0.8(60)=56\) points, and its variance is \(0.2(40-56)^2+0.8(60-56)^2=0.2(256)+0.8(16)=64\) points squared. Using the quantile definition \(Q(q)=\inf\{y:F(y)\geq q\}\), its 10th percentile is 40 and its median and 90th percentile are 60.
Now suppose candidate B instead assigns dropout probabilities 0.4 at \(Y=40\) and 0.6 at \(Y=60\). The overall dropout probability remains \(0.5(0.4)+0.5(0.6)=0.5\), but \(P(Y=60\mid D=1)=(0.6)(0.5)/0.5=0.6\). The implied mean is \(0.4(40)+0.6(60)=52\) points, and the variance is \(0.4(40-52)^2+0.6(60-52)^2=0.4(144)+0.6(64)=96\) points squared. Its 10th percentile is 40 and its median and 90th percentile are 60. The quantiles happen to match, but the means and variances differ.
The calculation verifies the probability reweighting in the selection-model formula. It also illustrates why a quantile comparison alone may miss differences between distributions. These values are hypothetical implications of the stated outcome and dropout probabilities, not observed missing outcomes.
Diagnostics, Alternatives, and Limitations
A table of means, variances, and quantiles is useful, but it is not a substitute for checking what was compared. Before interpreting a discrepancy, verify the following:
- Same target: Each candidate refers to the same outcome time, scale, dropout definition, conditioning history, and target group.
- Same pooling rule: Pooled summaries use the same participant weights or pattern proportions across candidates.
- Valid predictive distributions: Predicted values are on a meaningful scale, probabilities sum to one, variances are nonnegative, and quantiles are ordered.
- More than one summary: A similar mean can coexist with different spread, skewness, or tails; inspect several quantiles or a distribution plot as well.
- Numerical stability: If predictions come from simulation, use enough draws to estimate tail quantiles reliably and check that repeated runs do not materially alter the displayed summaries.
Plotting conditional distributions or quantile curves can reveal features that pooled summaries conceal, such as a model difference limited to one observed-history group. Conversely, a plot can make small differences look prominent if axes or scales differ. Keep axes and units common when placing candidates side by side, and distinguish a distribution of model predictions from a plot of observed outcomes.
One alternative is to compare candidates using agreement with the observed data alone. That addresses observed-data fit, not the distribution of outcomes after dropout. Another is to focus on a particular model parameter or a single summary such as the mean; that may be useful for a specific question but can discard important distributional differences. Distributional summaries add detail, but they do not resolve the fundamental limitation: the missing outcomes are unavailable for direct validation. A candidate's implied distribution depends on its assumptions about outcome and dropout, and observed-data agreement alone does not determine which extrapolation is correct.
A common mistake is to read a lower predicted mean as evidence that dropouts truly had lower outcomes. The correction is that the mean is conditional on the candidate model and its assumptions; it is not measured in the dropout group. Another is to conclude that two candidates agree because their means are close. As the examples show, their variances, quantiles, or subgroup distributions can still differ. Report the summaries as model-implied predictions and explain which parts of the distribution disagree.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A candidate model predicts a post-dropout outcome with mean 30 points and variance 25 points squared. What is its standard deviation, and what does the mean describe?
- Two candidates predict the same pooled mean for a dropout group. Give one reason their pooled variances could differ.
- In the discrete example, what would candidate A's implied dropout distribution be if dropout probability were 0.8 at \(Y=40\) and 0.2 at \(Y=60\), with prior probabilities still equal?
- Why is a model-implied 90th percentile not a directly observed description of outcomes after dropout?
Solutions. (1) The standard deviation is \(\sqrt{25}=5\) points. The mean is the average outcome predicted under that candidate for the stated post-dropout target group; it is not an observed mean among people with missing outcomes. (2) The candidates may have different within-group variances, different separation between group-specific means, or both. (3) The overall dropout probability is \(0.5(0.8)+0.5(0.2)=0.5\). Thus \(P(Y=40\mid D=1)=(0.8)(0.5)/0.5=0.8\), and \(P(Y=60\mid D=1)=0.2\). (4) The 90th percentile is calculated from the candidate's distribution for unobserved outcomes. Because those outcomes were not recorded after dropout, the percentile is a model-based implication rather than an empirical percentile computed from observed dropout outcomes.