Locating Timepoints and Subgroups With Model Disagreement

Why Locate Disagreement?

A single summary of disagreement across candidate missing-not-at-random (MNAR) models can hide where their assumptions matter. Models may give similar predictions early in follow-up but diverge later, or agree overall while producing quite different predictions for one prespecified subgroup. For a treatment comparison, the arms can each show substantial model-specific movement while the treatment contrast remains comparatively stable. Locating these patterns helps focus sensitivity interpretation on the follow-up times and populations where conclusions are most affected.

The task here is not to decide which candidate is best. It is to compare their predictions for the same estimand at specified times and in specified subgroups. Earlier material on model envelopes for sensitivity conclusions describes an overall summary; the time- and subgroup-specific summaries below show where that spread occurs.

Make the Comparisons Like-for-Like

These summaries are meaningful only if the candidate models are being asked the same question. Before calculating them, specify the outcome and its units, the follow-up times, treatment coding, subgroup definitions, and target population. Each \(\hat{\mu}_{k,a,g,t}\) must refer to that same combination. If models use different covariate distributions for standardization, for example, their predictions can differ because of both the MNAR assumptions and the target populations. Use common standardization when the goal is to isolate model sensitivity.

The subgroup should be defined before inspecting where models disagree, and the comparison should use the same subgroup definition in every model. A subgroup-specific mean can be estimated directly for that subgroup or standardized to a specified covariate distribution within it; state which interpretation is intended. Subgroups with little observed support can yield unstable estimates and extrapolations. A large spread in such a subgroup is a reason for caution, not by itself evidence that the subgroup truly behaves differently.

The models should also use a common outcome scale and comparable treatment estimands. For instance, a difference in outcome means is in outcome units, whereas a ratio is unitless; their spreads cannot be compared directly without a justified common scale. If a contrast is interpreted causally, its causal interpretation additionally requires the design and identification assumptions appropriate to the study, such as valid treatment assignment in a randomized trial. The calculation of model spread alone does not establish causality.

Prediction Spread and Treatment-Contrast Spread

At each time and subgroup, first inspect treatment-specific prediction spreads \(S_{g,t,a}\). Then inspect the contrast spread \(C_{g,t}\) if the scientific question concerns the difference between treatments. These are different summaries. Prediction spread describes how much the candidate models move a treatment-specific mean; contrast spread describes how much they move the difference between two means.

To justify the bound, write \(x_k=\hat{\mu}_{k,1,g,t}\) and \(y_k=\hat{\mu}_{k,0,g,t}\). For any two candidate models \(i,j\), $$ |(x_i-y_i)-(x_j-y_j)|=|(x_i-x_j)-(y_i-y_j)|\leq |x_i-x_j|+|y_i-y_j|. $$ The first absolute difference on the right is no greater than \(\max_k x_k-\min_k x_k=S_{g,t,1}\); the second is no greater than \(S_{g,t,0}\). Taking the largest left-hand difference over all \(i,j\) gives \(C_{g,t}\leq S_{g,t,1}+S_{g,t,0}\). The inequality explains why arm-specific results should accompany a treatment-contrast summary when interpretation depends on both.

Locate the Largest Practical Differences

A useful analysis lays out the summaries across the prespecified times and subgroups, rather than reporting only the largest number. A table or heat map can display \(C_{g,t}\), with arm-specific \(S_{g,t,a}\) available alongside it. Interpret the largest values against a practical threshold on the relevant scale, not merely by their rank. If outcomes use different scales across analyses, raw spreads cannot be compared as though they had a common meaning.

The following hypothetical comparison uses a prespecified 2.5-point threshold for practically important disagreement in the treatment contrast. The values are model-specific contrasts in outcome points; the spread in each cell is the maximum minus the minimum across the three candidate models.

Uncertainty, Diagnostics, and Limitations

Across-model spread and sampling uncertainty answer different questions. \(C_{g,t}\) describes variation in point estimates across the chosen MNAR assumptions. A model-specific standard error or interval describes uncertainty conditional on one model and its assumptions. Neither substitutes for the other. A small across-model spread does not guarantee precise estimates, and a large spread does not quantify the probability that any model is correct.

Where uncertainty is important, show model-specific intervals or uncertainty summaries alongside the cross-model spread. If estimates are obtained by resampling, refit each candidate model within each resample and calculate the same time- and subgroup-specific summaries; this reflects estimation variability under the specified resampling procedure. For Bayesian fits, model-specific posterior summaries can be shown, but they remain conditional on each model and its prior. Do not treat the range of point estimates as a confidence interval or as a posterior interval.

Several diagnostics help reveal when a comparison may be misleading:

• Check the estimand. Confirm that each model predicts the same outcome, time, treatment contrast, subgroup, and target population.
• Inspect support. Small subgroups, few observed outcomes at late visits, or sparse treatment-by-subgroup cells can make estimates unstable and MNAR extrapolation especially influential.
• Review arm-specific predictions. Large shifts in both arms may cancel in the contrast; cancellation should be visible rather than mistaken for model agreement everywhere.
• Check sensitivity to the candidate set. The range only summarizes models included in the comparison. Adding or removing a plausible candidate can change its endpoints.
• Keep scales and thresholds explicit. A numerical spread has no automatic clinical meaning. State the outcome units and the basis for any practical threshold.
• Avoid post hoc searching. Choosing subgroups or times because they show the largest disagreement exaggerates how confirmatory that finding is. Label exploratory searches as exploratory and account for the multiplicity of comparisons in interpretation.

These summaries complement, rather than replace, checks of fit to observed data and reasoned assessment of the extrapolation assumptions. The observed data generally cannot determine the unobserved outcomes after dropout without additional assumptions. Therefore, a time-specific disagreement pattern is evidence of sensitivity to the candidate assumptions, not a test that identifies the true missingness mechanism.

Common Mistake and Practical Alternatives

A common mistake is to see a small contrast spread and conclude that all candidate models make similar predictions. That conclusion can be false: treatment-specific means may each have large spreads that cancel when subtracted. The correction is to report the contrast spread for the treatment question and the arm-specific prediction spreads for context. Conversely, large arm-specific spreads do not automatically imply a large treatment-contrast spread.

A single overall envelope is concise but can conceal a localized time or subgroup sensitivity. A time- and subgroup-specific display is more informative when those dimensions were scientifically specified, though it can become noisy when many sparse cells are examined. Model-fit criteria or observed-data diagnostics address other aspects of candidate models; they do not replace comparing the estimand-specific predictions. Model averaging can produce a combined summary under its own weighting choices, but it may obscure the extent and location of disagreement among assumptions. The appropriate display depends on the question, and no spread measure ranks candidates by truth.