Why Inspect Predictions, Not Just Model Fit?
Longitudinal g-computation uses fitted conditional outcome models to predict outcomes under specified treatment regimes. Those predictions may be evaluated at covariate histories and treatment values that are rare or absent in the observed data. A model can fit outcomes well where data are plentiful and still make unreliable predictions in these less-supported parts of the covariate space. This is outcome-model extrapolation: relying on the model to extend beyond the combinations of covariates and treatments that the data adequately represent.
This concern is related to, but not identical to, diagnosing positivity for a candidate regime. Positivity asks whether the regime-assigned treatment has support at relevant histories. Outcome-model diagnostics ask whether the fitted outcome relationships can be trusted at the particular histories and assigned treatments where predictions are needed. Limited treatment support is one reason predictions may require extrapolation; sparse covariate combinations or an overly restrictive model can also make predictions fragile. Neither a good observed-data fit nor a diagnostic plot establishes that the causal assumptions hold.
Where Extrapolation Enters G-Computation
For a final outcome \(Y\), a simple conditional outcome model might estimate \(m(H,A)=E(Y\mid H,A)\). A regime-specific prediction requires evaluating that model at \(A=g(H)\), then averaging predictions over the target population's histories. In longitudinal settings, the procedure is recursive: fitted conditional models are evaluated at regime-assigned treatments and at histories generated or averaged over in earlier steps. Thus, the relevant support is specific to each model and each step of the computation.
A practical audit starts with the actual prediction records or histories used by the g-computation procedure. For each model, compare the covariate-treatment combinations where predictions are requested with those present in the model's fitting data. Counts, ranges, cross-tabulations, and plots can reveal sparse regions. For continuous covariates, examine the joint distribution and distances to observed records; being inside each covariate's separate range does not guarantee support for their combination. Also inspect the predicted outcome values. Extreme or implausible predictions can signal model dependence, though plausible-looking predictions do not prove adequate support.
List the histories and treatment values at which each fitted outcome model is evaluated under the candidate regime.
For each requested combination, examine counts, ranges, joint covariate patterns, and whether similar records exist with the assigned treatment.
Look for extreme predictions and check whether reasonable alternative model forms materially change predictions in weakly supported regions.
Describe which portions of the regime-specific prediction depend on sparse or unsupported data; do not interpret a fit statistic as a validation of those predictions.
Worked Example: A Regime Prediction Beyond the Treated Covariate Range
Hypothetical data. Suppose \(X\) is a baseline severity score and \(A\) is a binary treatment. The outcome is measured in points. The fitted conditional mean model is \(\widehat m(X,A)=50+4X+12A\). In the fitting data, treated participants have severity scores from 2 to 4, while untreated participants have scores from 0 to 8. A candidate rule assigns treatment to a person with \(X=7\).
The regime prediction for that person is \(\widehat m(7,1)=50+4(7)+12(1)=50+28+12=90\) points. The overall observed range of \(X\) includes 7, but the treated data used to fit the treatment-specific part of the relationship include no scores above 4. The model is therefore being asked to predict a treated outcome at \(X=7\), beyond the observed treated range.
This is a support warning, not proof that 90 is wrong. The linear equation imposes a particular continuation of the outcome relationship from \(X=2\)–4 to \(X=7\) among treated people. The observed data described here do not directly check that continuation. A useful report would state that the regime's prediction for high-severity treated individuals relies on extrapolation, rather than presenting the numerical prediction as equally well supported across all severity levels.
Why Excellent Observed Fit Can Mislead
Observed-data fit evaluates predictions where outcomes were observed, under the treatments people actually received. A regime can instead require predictions for different treatment-history combinations. If the observed data contain little information about those combinations, residuals calculated on observed records may reveal little about the accuracy of regime predictions. This gap is fundamental: fit is assessed on the observed treatment assignment, while g-computation evaluates models under the regime assignment.
The following simple argument shows how perfect observed fit can coexist with different predictions at an unsupported combination. Suppose the observed records all satisfy \(A=X\), where both variables are binary, and their outcomes satisfy \(Y=10+5X\). Consider the family of conditional mean models \(\mu_c(X,A)=10+(5-c)X+cA\), for any real value \(c\). For every observed record, \(A=X\), so \(\mu_c(X,X)=10+(5-c)X+cX=10+5X=Y\). Each model therefore fits every observed outcome exactly. But for the combination \(X=0,A=1\), which is absent from the data, the prediction is \(\mu_c(0,1)=10+c\). As \(c\) varies, the observed fit remains unchanged while the unsupported-combination prediction changes. The data and fit alone cannot choose among these predictions.
In a real analysis, models are not generally this simple, and the observed treatment pattern may not be perfectly deterministic. The argument nevertheless explains why residual plots, \(R^2\), or low prediction error on observed records cannot establish accurate predictions for regime-assigned treatments in poorly supported regions. Such diagnostics are useful for evaluating fit on the observed data; they do not substitute for examining where the regime asks the model to predict.
Worked Example: Perfect Fit, Different Regime Predictions
Hypothetical data. Four records are observed: two with \(X=0,A=0,Y=10\), and two with \(X=1,A=1,Y=15\). A regime assigns \(A=1\) to everyone. The observed combinations are only \((X,A)=(0,0)\) and \((1,1)\); neither record with \(X=0\) received \(A=1\).
For the model family \(\mu_c(X,A)=10+(5-c)X+cA\), the fitted value at \((0,0)\) is \(10\), and at \((1,1)\) it is \(10+(5-c)+c=15\). Thus every value of \(c\) gives zero residual for all four observed records.
Under the regime, the prediction for a person with \(X=0\) is \(\mu_c(0,1)=10+c\). For example, \(c=0\) gives 10, while \(c=4\) gives 14. Both models fit the observed data exactly, but they disagree by 4 outcome units for this regime-assigned combination. The data provide no observed \((0,1)\) record to resolve that disagreement. The example illustrates why the regime prediction requires an assumption about how the outcome relationship extends beyond observed treatment-history combinations.
Longitudinal Support Must Be Checked at Each Step
In a longitudinal analysis, a covariate history can be relevant because it is used to predict a later outcome, even if its individual components each appear common in the data. The order of events matters: a history includes earlier treatment and covariate values, and a later model may be evaluated at a regime-assigned treatment conditional on that history. A marginal summary such as “most values of \(L_2\) are common” can conceal a sparse joint combination of \(L_1,A_1,L_2\).
Worked Example: A Sparse History in a Two-Visit Regime
Hypothetical data. Consider a second-visit outcome model using \(L_1,A_1,L_2,A_2\). In the 100 records used to fit that model, the observed counts for two histories are as follows:
| History before \(A_2\) | Records with \(A_2=0\) | Records with \(A_2=1\) | Total |
|---|---|---|---|
| \(L_1=0,A_1=0,L_2=1\) | 8 | 2 | 10 |
| \(L_1=1,A_1=1,L_2=1\) | 3 | 27 | 30 |
Suppose a candidate dynamic regime assigns \(A_2=1\) whenever \(L_2=1\). At the first listed history, the model's prediction under the regime uses \(L_1=0,A_1=0,L_2=1,A_2=1\). Only 2 records support this combination. At the second history, the same regime-assigned treatment is represented by 27 records. Both histories have \(L_2=1\), but the amount of support for the relevant joint combination differs substantially.
The counts do not by themselves determine whether the model is adequate, and a small count does not establish a specific direction of bias. They do identify where predictions may be sensitive to the assumed model form. An analyst could report these counts, inspect the fitted predictions at both histories, and compare reasonable model specifications. The first history deserves particular scrutiny because its regime prediction is informed by very few records with the assigned treatment. In a full g-computation, this check is repeated at every model evaluation needed by the regime.
Practical Diagnostics and Interpretation
No single diagnostic gives a universal cutoff between supported and extrapolative predictions. The useful evidence depends on the data structure, number of covariates, model, and target population. Diagnostics should make the relevant data-model mismatch visible rather than certify a model as correct.
- Tabulate joint combinations. For discrete histories, count records for the history and assigned treatment together. Marginal counts for each variable can conceal empty or sparse joint cells.
- Compare distributions by assigned treatment. For continuous covariates, compare the covariate distributions among observed recipients of each treatment at the relevant histories. Inspect ranges and tails, not just means.
- Examine prediction locations. Identify whether regime-generated histories lie near observed records supporting that model. A point can be within every variable's marginal range yet unusual as a joint combination.
- Inspect predicted values. Find extreme, implausible, or abrupt predictions and trace them back to the histories and treatments that produced them.
- Check model sensitivity where support is weak. Compare predictions under reasonable alternative specifications. Large changes concentrated in sparse regions indicate that the regime estimate depends on modeling choices there.
Sensitivity to model form is a warning signal, not a guarantee that the least sensitive specification is correct. Conversely, similar estimates from several models do not create information where the data provide little support: models may share the same restrictive assumptions. Make the limitation explicit and distinguish data-supported predictions from predictions whose reliability depends heavily on extrapolation.
Common Mistake
Check Your Understanding
Answer each question by referring to the prediction task and its supporting data.
- A treated group's observed severity scores range from 1 to 5. A candidate regime requires a treated-outcome prediction at severity 8, although the overall severity range is 0 to 10. What support concern does this raise?
- Why can a model have zero residuals on observed records and still give uncertain predictions under a regime?
- In the two-visit example, why is the count of 27 records not an adequate substitute for checking the separate history with only 2 records?
- Can a predicted outcome that falls within a clinically plausible range establish that it is well supported by the data? Explain.
- Name two diagnostics that can help reveal extrapolation in a longitudinal g-computation analysis, and state what each examines.