Diagnostics for Parametric Longitudinal Models

Why Diagnose the Conditional Models?

The parametric g-formula relies on models for the successive covariate transitions and the outcome, conditional on the histories that precede them. A model can produce a plausible marginal estimate while fitting important parts of those conditional distributions poorly. Diagnostics ask a more local question: does the fitted model describe the observed data at each step of the longitudinal process?

Write \(H_t\) for the observed history available immediately before a variable is generated at time \(t\). It may include baseline covariates, earlier covariates, and treatment history. For a binary covariate \(L_t\), for example, the fitted model gives \(\hat p_t(H_t)=\widehat P(L_t=1\mid H_t)\). For a continuous variable, it gives a fitted conditional distribution \(\hat F_t(\,\cdot\mid H_t)\). Diagnostics compare these conditional predictions with the observed \(L_t\) or outcome among people with the corresponding observed histories.

This is different from checking a counterfactual estimand directly. The observed data contain outcomes under received treatment, not outcomes under every intervention. A good fit to observed conditional distributions is useful, but it does not by itself establish exchangeability, positivity, or correctness of extrapolation to histories rarely or never observed.

Residuals and Their Reference Properties

These quantities have different uses. Raw residuals can reveal systematic overprediction or underprediction. Standardized residuals put errors on a common scale, while PIT values assess the whole conditional distribution rather than only its mean. Binary residuals cannot form a smooth bell-shaped histogram: their possible values depend on the predicted probability. For binary variables, calibration summaries and proper prediction scores are usually more informative than a normality check.

Proof. Since \(m(H)\) is a function of \(H\), conditional expectation gives \(E[R\mid H]=E[X-m(H)\mid H]=E[X\mid H]-m(H)=0\). Taking expectations of both sides and applying the law of total expectation yields \(E[R]=E\{E[R\mid H]\}=0\). The integrability assumption ensures these expectations exist. \(\square\)

This result motivates plotting residuals against fitted values, time, and important components of history. A trend in their conditional averages may signal a missing nonlinear term, an interaction, or another systematic feature. A zero overall average is not enough: positive and negative errors in different history groups can cancel.

Calibration and Distributional Checks

For a binary variable \(X\), calibration asks whether observations with predicted probability near \(p\) occur with frequency near \(p\). Under a correct conditional mean \(m(H)=P(X=1\mid H)\), the identity \(E[X\mid m(H)]=m(H)\) follows by conditioning first on \(H\) and then on \(m(H)\). In practice, predictions are grouped into bins or summarized with a smooth calibration curve. Bins should not be so narrow that they contain too few observations to estimate event frequencies reliably.

The Brier score, \(n^{-1}\sum_i(X_i-\hat p_i)^2\), summarizes squared probability error; lower is better on the same evaluation sample and for the same outcome prevalence. It does not identify the source of poor fit, so it should accompany calibration plots or subgroup summaries. For continuous outcomes, inspect residuals against fitted values and history, their spread across time or predicted values, and whether the assumed conditional distribution describes tails and interval coverage.

Proof. Fix \(u\) with \(0<u<1\), and define the conditional quantile \(q_u(H)=\inf\{x:F(x\mid H)\geq u\}\). Continuity of the conditional CDF implies \(F(q_u(H)\mid H)=u\). Thus, conditional on \(H\), \(P(U\leq u\mid H)=P(X\leq q_u(H)\mid H)=F(q_u(H)\mid H)=u\). This conditional probability is the same for every \(H\); taking expectations gives \(P(U\leq u)=u\). The endpoints follow by limits. \(\square\)

The implication is one-way as a diagnostic: a uniform-looking PIT histogram is compatible with a correct conditional distribution, but a small sample can make substantial departures hard to see. A uniform marginal histogram also does not guarantee that the distribution is correct within important history groups. For discrete variables, the ordinary PIT is not continuous-uniform; randomized PIT values, formed by drawing uniformly between the CDF values immediately below and at the observation, are a possible distributional check.

Observed Versus Simulated Summaries

A complementary check compares observed summaries with predictions or trajectories generated from fitted models. For a one-step check, retain each person’s observed history \(H_t\), calculate the fitted conditional distribution at that history, and compare observed summaries with their model-based expectations. This isolates a conditional model from errors that might accumulate when earlier variables are also simulated.

For a trajectory check, simulate sequentially from fitted covariate and outcome models, using the observed baseline distribution and a clearly specified treatment process or treatment sequence. Compare summaries such as time-specific covariate prevalence, outcome risk, means, quantiles, and selected joint-history frequencies. If Monte Carlo simulation is used, simulation variability must be distinguished from discrepancies between data and model predictions.

Diagnostics in Practice: Interpretation and Failure Modes

No single diagnostic is a pass-or-fail test. Use plots and summaries that target different features, and report the scale of the comparison. For example, compare observed and predicted means by time and by substantively important history groups; check calibration for binary transitions and outcomes; and examine continuous-model residual spread, tails, and interval coverage. When observations from the same person contribute at multiple times, do not treat all residuals as independent evidence. Subject-level clustering and parameter estimation affect uncertainty around summaries.

A frequent mistake is to declare a model adequate because its residuals average to zero. The mean-zero result concerns residuals conditional on the full history under the true conditional mean. A marginal average near zero can conceal opposite errors in subgroups; it says nothing by itself about variance, tails, or time dependence. Another mistake is to interpret successful observed-data prediction as proof that counterfactual predictions are reliable. Under an intervention, simulated histories may differ from those common in the observed data, so a model can be asked to extrapolate beyond the support on which these diagnostics provide evidence.

When diagnostics flag a concern, first identify which variable, time, and history region are involved. Residual trends may suggest a functional-form issue; poor binary calibration may indicate probability errors; poor PIT or interval coverage may reveal distributional misspecification. Sparse data can make any of these checks inconclusive. Diagnostics are evidence about fit, not a mechanical rule for choosing a model, and they cannot verify untestable causal assumptions.