Comparing Predictions Across Dropout Patterns

Why Compare Predictions Within Dropout Patterns?

Two candidate MNAR models can fit the observed data similarly and still imply different outcomes for people who leave a longitudinal study. An overall average can obscure this: one model may predict much lower outcomes for people who drop out early but similar outcomes for people who remain. Comparing predictions within observed dropout patterns makes that distinction visible. It asks where the models disagree, rather than trying to decide from one summary which model is correct.

A dropout pattern is a subject’s observed sequence of visits. For the monotone dropout setting, patterns can be labelled by the last visit observed: for example, $g=1$ means observed at visit 1 but not later, and $g=T$ means observed through visit $T$. Let $G_i$ denote subject $i$’s pattern, $Y_{it}$ the outcome at visit $t$, and $O_i$ the outcomes and covariates actually observed for that subject. Candidate model $m$ specifies a distribution for the outcomes, including outcomes after dropout.

Definition: For pattern $g$, visit $t$, and candidate model $m$, define the pattern-specific predicted mean as $$ \hat{\mu}_{mgt}=\frac{1}{n_g}\sum_{i:G_i=g}\hat{E}_m(Y_{it}\mid O_i,G_i=g), $$ where $n_g$ is the number of subjects in pattern $g$. When $t$ is after the last observed visit for pattern $g$, this is an extrapolated mean: it depends on the candidate model’s assumptions about unobserved outcomes. A pairwise model disagreement is $\hat{\Delta}_{m,m',g,t}=\hat{\mu}_{mgt}-\hat{\mu}_{m'gt}$, measured in the outcome’s units.

The definition uses each pattern’s actual observed subjects and available histories. The predictions must be formed using information available for those subjects; a post-dropout outcome cannot be used as a predictor for that subject. For a fair comparison, candidate models should address the same outcome, visits, cohort, pattern definitions, and prediction target. Their observed-data components should be aligned as far as the scientific comparison permits. A difference in $\hat{\mu}_{mgt}$ then reflects a difference in model-based predictions, not a change in who is being summarized.

Comparing Predictions at a Common Target

A direct comparison computes each model’s predicted mean for the same pattern and visit, then takes their difference. A positive difference means model $m$ predicts a higher mean than model $m'$; a negative difference means it predicts a lower mean. Report the pattern, visit, outcome units, and whether the outcome is observed or extrapolated. For a visit after dropout, the comparison concerns model-based missing-outcome predictions, not observed measurements.

Differences between patterns need more care. Patterns can have different distributions of baseline covariates or observed histories. Thus, a contrast between their raw pattern-specific means can reflect both the patterns’ different compositions and the model’s predictions. If the goal is to compare predictions under a common covariate distribution, standardize each pattern’s predictions to the same target weights. State those weights; standardization changes the estimand and does not make the observed patterns interchangeable.

Worked Example: Finding the Largest Difference at a Dropout Visit

Suppose a hypothetical study measures a symptom score from 0 to 10 at three visits. Among 100 subjects, 20 are observed only at visit 1, 30 are observed through visit 2, and 50 complete visit 3. Two candidate MNAR models have the following predicted visit-3 pattern means. The visit-3 outcomes for the first two patterns are missing.

Pattern	Number of subjects	Model A predicted mean	Model B predicted mean
Observed through visit 1	20	4.8	3.9
Observed through visit 2	30	5.5	5.2
Observed through visit 3	50	6.2	6.2

For the first pattern, the model difference is $4.8-3.9=0.9$ score points, with Model A predicting the higher mean. For the second it is $5.5-5.2=0.3$ points, again with Model A higher. For completers it is $6.2-6.2=0$. The largest disagreement occurs for subjects observed only through visit 1: the models differ most on the most extended extrapolation.

For reference, the pattern-size-weighted predicted means across the cohort are $0.20(4.8)+0.30(5.5)+0.50(6.2)=5.71$ for Model A and $0.20(3.9)+0.30(5.2)+0.50(6.2)=5.44$ for Model B. The overall difference is $5.71-5.44=0.27$ points. That modest average difference does not remove the 0.9-point disagreement in the earliest-dropout pattern. These are predictions under hypothetical models, not observed visit-3 averages for all subjects.

Why the Pattern-Weighted Summary Works

When patterns partition the cohort and every subject has a prediction for the same visit, the pooled predicted mean is the weighted average of the pattern-specific predicted means. This is an arithmetic identity, not a claim that any candidate model is correct.

\hat{\mu}_{mt} =\frac{1}{N}\sum_{i=1}^{N}\hat{E}_m(Y_{it}\mid O_i,G_i) =\sum_g\frac{n_g}{N}\hat{\mu}_{mgt}.

To see this, split the sum over subjects into disjoint groups indexed by $g$. Within group $g$, the sum of predictions is $n_g\hat{\mu}_{mgt}$ by the definition of the pattern-specific mean. Dividing the sum of those group totals by $N$ gives the displayed result. Subtracting this identity for two models shows that their overall predicted-mean difference is the pattern-size-weighted average of their pattern-specific differences. Consequently, positive and negative differences can cancel in the overall summary.

This identity requires the same cohort, pattern weights, visit, and prediction target for both models. If a pattern has no subjects, its empirical pattern mean is undefined. If candidate models use different target populations or different weights, their overall summaries are not directly comparable by this decomposition.

Worked Example: Standardizing Across Different Pattern Compositions

Consider two hypothetical dropout patterns and a binary baseline covariate, $X$, classified as low or high. At a post-dropout visit, the predicted means are as follows. In pattern 1, 75% of subjects have low $X$; in pattern 2, 25% have low $X$. The remaining subjects in each pattern have high $X$.

Pattern	Model	Low $X$ prediction	High $X$ prediction
1	A	4	8
1	B	3	7
2	A	5	9
2	B	4	8

Using each pattern’s own covariate mix, the pattern-1 mean is $0.75(4)+0.25(8)=5$ under Model A and $0.75(3)+0.25(7)=4$ under Model B. For pattern 2, the corresponding means are $0.25(5)+0.75(9)=8$ and $0.25(4)+0.75(8)=7$. Each model difference is 1 score point.

Now standardize both patterns to a common target with equal proportions of low and high $X$. Pattern 1 then has means $0.50(4)+0.50(8)=6$ under A and $0.50(3)+0.50(7)=5$ under B. Pattern 2 has means $0.50(5)+0.50(9)=7$ under A and $0.50(4)+0.50(8)=6$ under B. Under this common target, the pattern-2 minus pattern-1 contrast is 1 point in each model. The unstandardized contrast was $8-5=3$ points under A and $7-4=3$ under B. Both calculations are valid for their stated targets; the standardized contrast is the one that holds this covariate mix fixed.

The models differ by 1 point in both patterns in this example, while their predicted levels vary by pattern. A comparison should not confuse a model difference with a pattern difference, or conceal which covariate distribution defines the reported means.

Locating Disagreement and Assessing Its Stability

A useful summary is a table or plot of $\hat{\mu}_{mgt}$ across patterns and visits, accompanied by pairwise differences. For a set of candidate models $\mathcal{M}$, the pointwise range

D_{gt}=\max_{m\in\mathcal{M}}\hat{\mu}_{mgt} -\min_{m\in\mathcal{M}}\hat{\mu}_{mgt}

measures the largest separation in predicted means at pattern $g$, visit $t$. It retains the outcome units, making it easy to relate the disagreement to a meaningful change in the outcome. The range identifies the most divergent location but does not indicate which model is scientifically preferable.

Worked Example: Ranking Disagreement Across Patterns

Three hypothetical models predict a score at visit 4 for two patterns. For early dropouts the means are 4.6, 3.8, and 4.2; for later dropouts they are 5.2, 5.0, and 5.1.

For early dropouts, $D_{g,4}=\max(4.6,3.8,4.2)-\min(4.6,3.8,4.2)=4.6-3.8=0.8$ points. For later dropouts, $D_{g,4}=5.2-5.0=0.2$ points. Thus the greatest point-estimate disagreement is in the early-dropout pattern. If that pattern contains only 8 subjects while the later pattern contains 42, the 0.8-point range is also based on a smaller group and may be less stable. Pattern size alone does not quantify uncertainty, but it is an immediate reason to inspect it.

When parameter uncertainty is available, calculate predictions for each model across parameter draws or bootstrap refits, then summarize uncertainty for each pattern-specific mean and difference. The spread of predictions across candidate models and the uncertainty within each model answer different questions: the first describes sensitivity to model specification, while the second describes uncertainty conditional on a model and its fitting procedure. Neither should be substituted for the other.

Assumptions, Diagnostics, and Limitations

Pattern-specific predicted means are well-defined only when the pattern and visit are specified consistently, the model supplies a finite prediction for each relevant subject, and the target population and weighting rule are clear. Their interpretation as predictions for missing outcomes additionally relies on each candidate model’s assumptions for the unobserved outcomes and on the observed-data information used to fit it. The predictions cannot, by themselves, establish that those assumptions hold: the missing outcomes are not available for direct comparison.

Check the following before interpreting a comparison:

Pattern definitions: verify that each model uses the same visit schedule and dropout classification. A label such as “early dropout” must refer to the same observed history in every model.
Common target: confirm that the same subjects, covariates, visits, and standardization weights define each comparison. If patterns are standardized, report the target weights.
Observed versus extrapolated visits: mark which pattern-time cells contain observed outcomes and which are predictions after dropout. Model disagreement in an extrapolated cell is sensitivity to assumptions, not an observed-data discrepancy.
Support and precision: report pattern counts and inspect whether predictions depend on sparse covariate-history combinations or extreme estimates. A numerical mean can be computable but unstable.
Scale and plausibility: check that predicted means lie on a meaningful outcome scale and that the model’s predicted distributions do not imply impossible values. A plausible mean alone does not guarantee a plausible distribution.

Comparing within-pattern predictions complements, rather than replaces, other comparisons. Observed-data fit asks how well candidate models reproduce observed information; it does not settle their differing extrapolations. Comparing overall predicted means summarizes the cohort but can hide offsetting pattern-specific differences. Post-dropout trajectory plots show how predictions evolve over time, while this comparison emphasizes which observed dropout groups account for the disagreement. Treatment-effect comparisons address a different target: similar pattern-specific means do not guarantee similar treatment effects, and a difference in predicted outcome levels is not itself a treatment effect.

A common mistake is to rank models by the largest predicted mean or declare the model with less pattern variation “better.” The correction is to treat these displays as comparisons of implications under candidate assumptions. Large divergence locates an important sensitivity, but it does not identify the true model. Conversely, close predictions may simply reflect shared assumptions or limited information. Avoid ranking models from these summaries alone; model assessment requires evidence and criteria suited to the question.

Key takeaway: Compare candidate models at the same visit, within the same observed dropout pattern, and on the same target population. Report pattern-specific differences before averaging: an overall agreement can conceal substantial disagreement in the outcomes most dependent on extrapolation.

Check Your Understanding

Answer each question, then compare your reasoning with the short solutions.

In a pattern of 12 subjects, two models predict post-dropout means of 6.1 and 5.4 symptom points. What is the signed difference if Model A is listed first, and what does it mean?
Two pattern-specific model differences are 1.2 points for a pattern containing 10% of the cohort and $-0.4$ points for a pattern containing 90%. What is their pattern-weighted overall difference?
Why might raw pattern means be unsuitable for comparing patterns when baseline covariate distributions differ?
A model range is 0.7 points in a pattern with 6 subjects and 0.5 points in a pattern with 60 subjects. Can the first range be declared statistically meaningful from these numbers alone?
What does a large difference between candidate models for an unobserved visit establish, and what does it not establish?

Solutions. (1) $6.1-5.4=0.7$ points. Model A predicts a mean 0.7 symptom points higher than Model B for that pattern; the contrast is extrapolated. (2) $0.10(1.2)+0.90(-0.4)=0.12-0.36=-0.24$ points. The weighted difference is $-0.24$ points, even though one pattern has a positive difference. (3) The means can reflect both different covariate compositions and different pattern-specific predictions. A common standardization target separates the comparison from that composition difference. (4) No. These point-estimate ranges contain no uncertainty information; the smaller pattern may be less stable, but uncertainty requires an appropriate calculation. (5) It establishes that the candidate models imply different predictions under their assumptions at that pattern and visit. It does not establish which prediction is correct or that either model is supported by the unobserved outcomes.