Why Check Fit Separately by Pattern?
A pattern-mixture analysis describes outcomes conditional on dropout pattern. A model can reproduce an overall average while representing one pattern poorly, especially when patterns have different outcome distributions or sizes. Checking fit within each pattern therefore asks a focused question: does the fitted distribution reproduce the observed outcomes for the people in this pattern, at the occasions observed for them?
These checks are diagnostics, not a proof that a model is correct. The observed data can inform the distribution of outcomes that are observed in a pattern. They cannot, by themselves, verify the model's distribution for outcomes missing in that pattern or justify an assumption used to identify those outcomes. The aim here is to find visible disagreements between observed and fitted distributions before relying on model-based summaries.
The comparison must match the model's target. A marginal distribution within a pattern is not the same as a conditional distribution at fixed covariates. Nor does matching each occasion's marginal distribution imply that the model captures dependence between repeated outcomes. Those are separate questions: state which distribution is being checked, and do not draw broader conclusions than the check supports.
Empirical Summaries as First Checks
Begin with summaries that are interpretable in the outcome's units: sample size, mean and standard deviation for quantitative outcomes, selected quantiles, and proportions or category counts for discrete outcomes. Compare each to the corresponding fitted quantity. For example, a fitted mean can be compared with the observed mean; for a fitted distribution \(\widehat F_{gj}\), its \(p\)th quantile is \(\widehat F_{gj}^{-1}(p)\), where \(p\) is a probability such as 0.25, 0.50, or 0.75.
These summaries are useful because they point to types of mismatch. A close mean but a much larger observed standard deviation suggests a spread problem. A close median but a disagreement in an upper quantile suggests that the upper tail may be poorly represented. For categorical outcomes, compare observed category proportions with fitted category probabilities. For binary outcomes, the observed proportion is one summary, but when fitted probabilities vary across individuals, compare it with the average fitted probability rather than with an arbitrarily selected individual probability.
Worked Example: Comparing Mean, Spread, and Median
Suppose six hypothetical observed outcomes in one dropout pattern, measured in units, are \(4, 5, 5, 6, 7, 9\). A fitted normal distribution for this pattern and occasion is \(N(6,4)\), where the second parameter is the variance. Compare the observed summaries with the fitted distribution.
Observed mean and spread. The observed mean is \(\bar{x}=(4+5+5+6+7+9)/6=36/6=6\) units. The squared deviations from 6 sum to \(4+1+1+0+1+9=16\), so the sample standard deviation is \(s=\sqrt{16/(6-1)}=\sqrt{3.2}\approx1.789\) units. The fitted normal distribution has mean 6 units and standard deviation \(\sqrt{4}=2\) units. The means agree, while the observed sample spread is somewhat smaller.
Observed and fitted median. With six observations, the empirical median is the average of the third and fourth ordered values: \((5+6)/2=5.5\) units. The fitted normal distribution is symmetric about 6, so its median is 6 units. These are close, but the difference is worth interpreting alongside the small sample size; six observations cannot provide a precise picture of the pattern's distribution.
This comparison does not establish that the normal model fits. It shows that the mean agrees and that the observed spread and median do not raise an obvious large discrepancy in this small example. A quantile plot and additional observations could reveal features these summaries miss.
Quantile Comparisons
A quantile comparison places ordered observed values beside quantiles predicted by the fitted distribution. For \(n_g\) observations in pattern \(g\), order the values as \(y_{(1)}\le\cdots\le y_{(n_g)}\). A common approximate plotting position for rank \(i\) is \(p_i=(i-0.5)/n_g\). Compare \(y_{(i)}\) with \(\widehat F_{gj}^{-1}(p_i)\). Plotting positions are conventions for arranging a diagnostic; they are not exact probabilities that the observations must equal.
If the fitted distribution represents the observed distribution reasonably well, the points should generally follow the equality line when observed quantiles and fitted quantiles use the same units. A systematic curve can indicate skewness not captured by the model. Points that diverge at one end can indicate a tail mismatch. One or two extreme points in a small pattern may be ordinary sampling variation, so consider the size of the pattern and the overall shape rather than treating any departure as decisive.
Worked Example: Quantiles Under a Lognormal Fit
Consider eight hypothetical positive outcomes in one dropout pattern, in units: \(2,3,4,5,7,9,12,30\). Suppose the fitted model is lognormal, defined here by \(\log(Y)\sim N(\log(6),0.8^2)\). Its quantile at probability \(p\) is \(6\exp(0.8z_p)\), where \(z_p\) is the standard normal quantile at \(p\). Compare the fitted and observed lower and upper endpoints using the plotting positions.
For rank 1, \(p_1=(1-0.5)/8=0.0625\), and \(z_{0.0625}\approx-1.534\). Thus the fitted quantile is \(6\exp(0.8(-1.534))\approx1.759\) units, compared with the observed first value of 2 units.
For rank 8, \(p_8=(8-0.5)/8=0.9375\), and \(z_{0.9375}\approx1.534\). The fitted quantile is \(6\exp(0.8(1.534))\approx20.47\) units, whereas the observed maximum is 30 units. The observed upper endpoint is notably larger than the fitted quantile at this plotting position. This is a warning to inspect the upper tail, not proof that the fitted distribution is wrong: the sample has only eight values, and its maximum is variable.
The observed median is \((5+7)/2=6\) units, matching the fitted median of 6 units. Together, the central quantile and endpoint comparisons suggest that the center is represented but the upper tail may not be. A quantile plot of all eight ranks would show whether the discrepancy develops gradually or is driven mostly by the largest observation.
Graphical Checks and a Short Justification
Useful displays include an empirical cumulative distribution function (ECDF) overlaid with the fitted cumulative distribution, and a quantile-quantile plot of observed against fitted quantiles. For discrete outcomes, use category-proportion plots or compare observed counts with fitted expected counts; a continuous ECDF can obscure the ties that are intrinsic to discrete data. Label the pattern and occasion clearly. When patterns have very different sample sizes, show the sample size because a similar-looking discrepancy has different evidential weight in a small and a large pattern.
The rationale for a quantile comparison follows directly from the definition of a quantile. If \(F\) is a continuous fitted CDF and \(Y\) truly follows \(F\), then \(F(Y)\) has a uniform distribution on \((0,1)\). Consequently, ordered observations correspond approximately to increasing probabilities, and the \(i\)th ordered observation can be compared with the fitted quantile at a plotting position near its rank.
The implication motivates the plot; it does not claim that every plotted point should lie exactly on the equality line. The ordered values vary from sample to sample, and fitted parameters are themselves estimated. If a model was fitted using the same observations being checked, its apparent fit can be optimistic. Where practical, assess stability with a held-out sample or resampling, while recognizing that small pattern-specific samples make splitting data especially costly.
Worked Example: Checking Binary Probabilities Within a Pattern
Suppose a fitted binary-outcome model gives hypothetical probabilities of 0.4 for five people and 0.8 for five others in one pattern. Assume outcomes are conditionally independent Bernoulli variables under this fitted model. The observed data contain two positive outcomes in the first group and five in the second, for a total of seven positives among ten people.
The average fitted probability is \((5(0.4)+5(0.8))/10=(2+4)/10=0.6\), so the fitted expected number of positives is 6 and the fitted average is 0.6. The observed proportion is \(7/10=0.7\). To describe the scale of variation under the stated independence model, the variance of the fitted total is the sum of the Bernoulli variances: \(5(0.4)(0.6)+5(0.8)(0.2)=1.2+0.8=2.0\). The standard deviation is \(\sqrt{2.0}\approx1.414\) positives. The observed total is one positive above expectation, or \(1/\sqrt{2}\approx0.707\) fitted standard deviations.
This is a modest discrepancy on that scale, not a formal goodness-of-fit conclusion. The calculation relies on the assumed conditional independence and the stated fitted probabilities. It also shows why comparing the observed proportion with one person's probability would be inappropriate: the relevant fitted average is 0.6.
Pattern-Specific Diagnostics in Practice
A disciplined check keeps the pattern-specific comparisons distinct before considering an overall summary. One practical sequence is:
State the pattern, occasion, outcome scale, and whether the comparison is marginal within the pattern or conditional on covariates.
Set observed means, spreads, quantiles, or category proportions beside the corresponding fitted quantities.
Use an ECDF, quantile plot, or discrete-category display suited to the outcome type, and look for systematic rather than isolated differences.
Note pattern sample sizes, possible parameter-estimation optimism, and that only outcomes observed in that pattern can be checked directly.
A pooled plot or pooled summary can be useful for a population-level descriptive question, but it is not a substitute for these pattern-specific checks. If one pattern has a high mean and another a low mean, pooling can conceal both discrepancies. Conversely, a pattern-specific discrepancy need not imply a poor marginal fit if that pattern has little weight in a particular population summary. The two checks answer different questions.
If covariates are included, the diagnostic also needs to respect the model's conditioning. For instance, a pattern may have a different covariate composition from another pattern. A mismatch between its raw observed mean and an unstandardized fitted mean could reflect a comparison of unlike targets rather than a distributional failure. Compare like with like: either evaluate predictions at observed covariate values and aggregate them appropriately, or examine observed and fitted distributions within relevant covariate strata.
Common Pitfalls, Alternatives, and Limits
- Wrong reasoning: “The pooled mean agrees, so every pattern fits.” Correction: calculate and inspect summaries within each pattern. Opposing pattern-specific errors can cancel in a pooled mean.
- Wrong reasoning: “A close mean proves the distribution fits.” Correction: the mean does not determine spread, skewness, tails, or category probabilities. Add quantiles and graphical checks.
- Wrong reasoning: “One extreme point proves model failure.” Correction: assess the full pattern, sample size, and sampling variability. An extreme observation is a reason to investigate, not an automatic rejection.
- Wrong reasoning: “Good fit to observed values verifies the missing outcomes.” Correction: outcomes not observed in a pattern cannot be directly checked with that pattern's observed data. Goodness of fit to observed values does not establish assumptions about the unobserved distribution.
Formal goodness-of-fit tests can supplement graphical and summary checks, but they are not a universal replacement. Their power and calibration depend on sample size, the fitted model, and whether parameter estimation is accounted for. With many patterns, small samples can make tests uninformative; with large samples, they can flag small discrepancies with little practical consequence. Graphical comparisons emphasize the location and form of disagreement, while summaries offer compact, interpretable contrasts. Neither approach alone captures every feature of a distribution.
Separate occasion-specific checks also do not establish that the joint distribution of repeated outcomes within a pattern is adequate. Two models can match each occasion's marginal distribution but differ in the dependence between occasions. Likewise, checks of observed outcomes do not validate causal interpretations or resolve assumptions about dropout. Treat these diagnostics as evidence about the observed-data fit of the specified pattern-specific distributions, and keep those limits explicit.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In one pattern, the observed mean matches the fitted mean, but the observed upper quartile is much larger than the fitted upper quartile. What feature should be investigated?
- For \(n_g=10\), what plotting position is assigned to rank 1 by \(p_i=(i-0.5)/n_g\)?
- A binary model gives fitted probabilities 0.2 and 0.8 for two people. Why is comparing the observed two-person proportion to 0.2 alone generally inappropriate?
- What can a good fit to observed outcomes in an early-dropout pattern establish about outcomes that are unobserved after dropout?
Solutions. (1) Investigate the upper part of the distribution, including possible skewness or a heavy upper tail; matching means alone does not settle distributional fit. (2) \(p_1=(1-0.5)/10=0.05\). (3) The relevant fitted expected proportion is the average fitted probability, \((0.2+0.8)/2=0.5\), not either person's probability by itself; the comparison also depends on the outcome model's assumptions. (4) It establishes no direct fact about those unobserved outcomes. Fit to observed values cannot verify the unobserved distribution or the assumptions used to identify it.