Model Checking for Joint Outcome-Dropout Fits

Why Check Both Parts of a Joint Fit?

A joint outcome-dropout model makes claims about two linked processes: the repeated outcomes while a subject is observed, and the timing of dropout. A model can reproduce one process reasonably well and still miss important features of the other. For example, it might fit the average outcome trajectory while predicting too few early dropouts. Checking both parts helps reveal such discrepancies before interpreting fitted parameters or using the model to describe unobserved outcomes.

These checks assess whether a fitted model can reproduce features of the observed data. They do not show that the model has correctly described outcomes that were never observed. That distinction is central: post-dropout outcomes are unavailable for direct residual checks, and the fit to observed records cannot by itself verify the extrapolation to them.

Use subject index \(i=1,\ldots,n\), visit index \(j\), outcome \(Y_{ij}\), and shared random effects \(b_i\). Let \(D_i\) denote the dropout time, with the convention for whether dropout occurs before or after a scheduled visit stated explicitly in an analysis. Write \(\theta\) for the model parameters. In the usual shared-parameter construction, the outcome and dropout processes are conditionally independent given \(b_i\) and the included covariates, while both depend on \(b_i\). The fitted joint model therefore predicts outcome and dropout features together, not as unrelated summaries.

Residual Checks for Repeated Outcomes

For a continuous outcome, a basic residual is the observed value minus its model-predicted value. In a repeated-measures model, the prediction must be defined carefully. A useful diagnostic is a one-step-ahead predictive residual: predict \(Y_{ij}\) using the model and the subject’s observed history through visit \(j-1\), then compare that prediction with the observed \(Y_{ij}\). The predictive distribution should account for uncertainty about \(b_i\), rather than treating an estimated random effect as known.

If the predictive mean and standard deviation are \(m_{ij}\) and \(s_{ij}\), respectively, the standardized predictive residual is

A well-calibrated set of predictive residuals should be centered near zero and should not show systematic trends against visit, fitted value, covariates, or prior outcome history. Their spread and tails should also be compatible with the predictive distribution. These are diagnostic expectations, not universal exact cutoffs: residuals from the same subject are dependent, and the model parameters are estimated from the data.

For a non-Gaussian outcome, a raw residual may not have a stable interpretation across fitted values. One option is a probability integral transform (PIT) value, the predictive cumulative probability assigned to the observed outcome. For a discrete outcome, randomization within the probability mass at the observed value avoids the ties that otherwise prevent a continuous uniform reference. A pattern of PIT values that departs from the model’s predictive distribution can indicate miscalibration.

A common diagnostic failure is to calculate residuals after substituting each subject’s fitted random effect and then treat them as ordinary independent residuals. Because those random effects are estimated using the subject’s own outcomes, the resulting residuals can look artificially small. Predictive checks that integrate uncertainty in \(b_i\), or compare full replicated data with the observed data, better reflect the uncertainty relevant to checking the joint model.

Checking Dropout Timing

For discrete-time dropout, let \(h_{ij}\) be the model’s probability that subject \(i\) drops out in interval \(j\), conditional on being at risk at its start and on the information used for that prediction. If \(S_{ij}\) is the set of subjects at risk at the start of interval \(j\), the expected number of dropouts in that interval is

Compare the observed number of dropouts in each interval with \(E_j\), and examine the overall shape of dropout timing, not only the total number of dropouts. For a continuous-time hazard, analogous checks can compare observed event-time summaries with replicated event times from the fitted model. In either setting, predictions should respect each subject’s risk period and the model’s conditioning information.

If the model uses observed outcome history to predict dropout, checks of dropout by history can be informative: for example, compare observed and predicted dropout counts among subjects with relatively high or low recent outcomes. Such comparisons assess whether the joint fit reproduces the observed association between outcome history and dropout. They do not establish that the same association holds for outcomes after dropout, which cannot be observed.

Posterior Predictive Comparisons

A posterior predictive check compares the observed data with data generated under the fitted Bayesian model. For each draw of parameters from their posterior distribution, generate shared random effects, outcomes, and dropout according to the joint model. Apply the same observation and dropout rules used in the study, then calculate the diagnostic summary for that replicated data set. This procedure preserves the model’s dependence between outcomes and dropout.

Let \(T(Y,D)\) be a chosen summary, such as the number of early dropouts or the proportion of observed outcomes above a clinically meaningful threshold. For each posterior predictive replicate, calculate \(T(Y^{\mathrm{rep}},D^{\mathrm{rep}})\) and compare its distribution with the observed value \(T(Y,D)\). The summary must be selected to target a feature of interest: a check of the overall mean, for instance, may miss excessive variability or an unusual tail.

The comparison may be two-sided in substance even when a one-sided tail area is calculated: values near either extreme can flag a mismatch. Posterior predictive tail areas are not uniformly distributed under a correctly specified model, in part because the observed data are also used to estimate the posterior. Treat them as descriptive diagnostics, alongside plots and the magnitude of the discrepancy, rather than as a pass-or-fail significance test.

What the Checks Can and Cannot Establish

A useful checking plan targets several distinct features rather than relying on one global score. The model should reproduce outcome distributions over time, including their location, spread, and relevant tail behavior; dropout timing and risk-set patterns; and observed associations between outcome histories and later dropout. Posterior predictive plots can show discrepancies that a single number hides. For example, similar total dropout counts can conceal excess early dropout balanced by too little late dropout.

Interpret an observed mismatch as evidence that the fitted model does not reproduce that particular feature well, not as proof of one unique cause. Possible explanations include an inadequate outcome distribution, a poorly represented dropout hazard, omitted covariates, dependence not captured by the shared effects, or influential observations. Model checking identifies questions for further investigation; it does not automatically determine which revision is correct.

Most importantly, observed-data checks cannot directly assess the distribution of outcomes after dropout. There are no observed residuals for those outcomes. A joint model supplies predictions there through its assumptions about the outcome process, dropout process, and their shared effects. Agreement with observed data is necessary evidence of adequacy for those data, but it cannot verify this unobserved part of the model. A good observed-data fit should therefore not be reported as proof that post-dropout predictions are correct.

The comparison with convergence diagnostics is important: convergence and boundary checks address computation and estimation, while model checks address how well the fitted model reproduces observed features. A converged fit can still show poor predictive behavior. Conversely, a conspicuous discrepancy does not by itself prove that the estimation algorithm failed. Keep these questions separate.

A Practical Checking Sequence

This sequence does not provide a mechanical model-selection rule. It makes the check interpretable and helps prevent a common error: treating a good match on one aggregate statistic as a general validation of the entire joint model.