Why Diagnose the Probability Models?
Longitudinal inverse-probability weights are built from fitted probabilities at successive visits. A probability model that predicts treatment or continued observation poorly can therefore affect many weights, not just one. Diagnosing these models means examining whether their fitted probabilities are compatible with the observed treatment and censoring patterns in the histories where those probabilities are used.
These checks address a specific question: does the fitted probability model describe the observed conditional probabilities adequately enough to support the intended weighting procedure? No single diagnostic answers that question, and good-looking diagnostics do not establish sequential exchangeability, consistency, or positivity. Those are causal identification conditions, not consequences of model fit.
The treatment model must provide a probability for the treatment actually received. Thus, for an individual with \(A_t=1\), the fitted denominator probability is \(\hat e_t(H_t)\); for \(A_t=0\), it is \(1-\hat e_t(H_t)\). For a censoring model written in terms of remaining observed, the corresponding probability for a person observed at the next time point is \(\hat q_t(H_t,A_t)\). Stabilized weights may use numerator probabilities from reduced-history models as well as these denominator probabilities. The diagnostics below primarily examine the denominator models, since their fitted probabilities enter the inverse factors that correct for treatment and censoring.
Calibration: Do Predicted Frequencies Match Observed Frequencies?
A model is calibrated in a group of observations if the average predicted probability in that group agrees with the observed proportion of events. For a treatment model, the event is receiving treatment; for a model of remaining observed, the event is remaining observed. One practical check is to group observations by fitted probability, then compare the mean fitted probability with the observed event proportion in each group.
For example, among people with predicted treatment probabilities near 0.30, roughly 30% would be expected to receive treatment if predictions were calibrated in that group. This is a comparison of predicted and observed frequencies, not a comparison of individual outcomes with deterministic predictions. Individual treatment assignments remain random.
Worked Example: Reading a Treatment Calibration Table
Suppose a hypothetical study has 150 eligible person-visits, divided into three groups of 50 according to the fitted probability of treatment. The table gives each group’s mean fitted probability and observed treatment count.
| Predicted-probability group | Mean fitted probability | Treated | Observed treatment proportion |
|---|---|---|---|
| Low | 0.10 | 5 of 50 | 0.10 |
| Middle | 0.40 | 17 of 50 | 0.34 |
| High | 0.80 | 43 of 50 | 0.86 |
In the low group, the observed proportion is \(5/50=0.10\), matching the mean fitted probability of 0.10. In the middle group, it is \(17/50=0.34\), which is 0.06 below the mean fitted probability of 0.40. In the high group, it is \(43/50=0.86\), which is 0.06 above the mean fitted probability of 0.80. The middle and high groups suggest some calibration error, although this small table alone does not establish whether the differences reflect systematic model misspecification or ordinary sampling variation.
The overall predicted number treated is \(50(0.10)+50(0.40)+50(0.80)=65\). The observed number is \(5+17+43=65\). Thus, the model gets the overall treatment frequency exactly right while still showing discrepancies within probability groups. Overall agreement is useful to report, but it can conceal errors in parts of the prediction range.
Calibration summaries depend on how observations are grouped. Narrow groups can reveal local discrepancies but may contain too few observations for stable observed proportions; broad groups can smooth away meaningful patterns. A plot of observed proportions against mean fitted probabilities, with group sizes and uncertainty made visible, is often more informative than a single calibration statistic. For censoring, apply the same logic to the probability of remaining observed, among the visits at which censoring could occur.
History-Specific Residual Patterns
For binary treatment, a simple residual is \(A_t-\hat e_t(H_t)\). If the true conditional probability is \(e_t(H_t)\), then its conditional mean is zero:
This gives a practical diagnostic idea: examine whether residuals show a systematic pattern across history variables that should predict treatment. A positive average residual in a history subgroup means more treatment occurred there than the model predicted on average; a negative average means less occurred. For a censoring model whose event is remaining observed, use \(R_{t+1}-\hat q_t(H_t,A_t)\) among eligible visits. Patterns may be examined across individual covariates, clinically meaningful combinations of history, visit time, and treatment history.
These are diagnostic comparisons, not a new causal estimand. A residual pattern can flag that the fitted conditional probabilities are inadequate, but failure to see a pattern does not prove the model is correct. In particular, an overall residual average near zero can result from positive and negative errors canceling across histories.
Worked Example: Overall Calibration Hides History-Specific Error
Consider 100 hypothetical eligible person-visits divided into two histories, \(H=1\) and \(H=0\). The fitted treatment model predicts a treatment probability of 0.75 for each of 40 people with \(H=1\), and 0.25 for each of 60 people with \(H=0\). Suppose 22 people in the first group and 23 in the second actually receive treatment.
For \(H=1\), the predicted number treated is \(40(0.75)=30\), while the observed number is 22. The residual sum is \(22-30=-8\), and the mean residual is \(-8/40=-0.20\). For \(H=0\), the predicted number is \(60(0.25)=15\), while the observed number is 23. The residual sum is \(23-15=8\), and the mean residual is \(8/60\approx0.1333\).
Across all 100 visits, the predicted total is \(30+15=45\), and the observed total is \(22+23=45\). The overall observed treatment proportion and overall mean fitted probability are both 0.45, so the overall residual sum is zero. Yet the opposite history-specific residual patterns are substantial: the model overpredicts treatment for \(H=1\) and underpredicts it for \(H=0\). A diagnostic that reports only the overall treatment frequency would miss this problem.
In practice, one would investigate whether the history groups correspond to important predictors or combinations of predictors, and whether the pattern recurs across visits. Sparse combinations require caution: a striking subgroup difference based on very few observations can be unstable. The goal is to find coherent conditional patterns, not to search indefinitely for isolated discrepancies.
Inspecting Predicted-Probability Distributions
For each visit, inspect the distribution of \(\hat e_t(H_t)\) and \(\hat q_t(H_t,A_t)\), preferably also separating observations by treatment actually received or by relevant history. Summaries such as quantiles, histograms, and the minimum and maximum can reveal whether predictions are concentrated in a narrow range or whether some observations have very small predicted probabilities for events that did occur.
A very small fitted probability for the treatment actually received produces a large inverse-probability factor. Likewise, a small fitted probability of remaining observed produces a large censoring factor for a person who remains observed. Such values may arise because the data contain histories with little support, because the model has fitted extreme predictions, or because the model is misspecified. Examining the distribution helps identify where to investigate; it does not by itself distinguish these explanations.
Check distributions visit by visit as well as across the full follow-up. A seemingly moderate overall distribution can conceal extreme fitted probabilities at one visit or in a particular history. Also distinguish treatment probabilities from probabilities of the observed treatment: for a person with \(A_t=0\), the relevant probability is \(1-\hat e_t(H_t)\), not \(\hat e_t(H_t)\). These checks complement positivity diagnostics, which focus on whether treatment options have adequate support across histories.
Worked Example: Tracing Fitted Probabilities Into a Cumulative Weight
Consider a hypothetical person observed at two treatment visits and remaining observed through the next time point at each visit. Suppose the person receives \(A_1=A_2=1\). At visit 1, the treatment numerator and denominator probabilities for the treatment received are 0.50 and 0.80. At visit 2, they are 0.50 and 0.70. The censoring numerator and denominator probabilities of remaining observed are 0.95 and 0.90 at visit 1, then 0.90 and 0.60 at visit 2.
The two treatment-weight contributions, calculated from the supplied probabilities, are
The two censoring-weight contributions are
The combined stabilized treatment-and-censoring weight is therefore
or 0.7068 to four significant figures. The visit-2 censoring denominator of 0.60 contributes a factor \(0.90/0.60=1.5\), larger than 1, because remaining observed was less likely under the denominator model than under the numerator model. The resulting combined weight is nevertheless below 1 because the treatment contributions together are smaller than 1. This example shows why a diagnostic should trace unusual contributions through the full product: an extreme factor at one visit can be offset by other factors, so the final weight alone does not reveal which fitted model deserves investigation.
A Practical Diagnostic Sequence
Diagnostics are most useful when performed in the order of the probability models and their role in the weight. The following sequence helps connect model checks to the resulting cumulative weights without treating any one check as definitive.
For each visit, specify who is eligible for treatment and who is at risk of censoring. Confirm whether each fitted probability represents receiving treatment, receiving the alternative treatment, or remaining observed.
Assess calibration by visit and across meaningful ranges of fitted probability. Report group sizes so that apparent discrepancies can be judged in light of how much information each group contains.
Summarize \(A_t-\hat e_t(H_t)\) or \(R_{t+1}-\hat q_t(H_t,A_t)\) across important observed histories. Investigate coherent patterns rather than relying on a zero overall residual average.
Review fitted-probability distributions, identify small probabilities for events that occurred, and trace their inverse factors into the cumulative weights by visit and model component.
The models are fitted rather than known, so their diagnostic results are also affected by estimation. In-sample diagnostics can look favorable because the same observations were used to fit the models. When feasible, predictions evaluated on data not used to fit the model, or through cross-validation, can provide a more demanding check of predictive calibration. These checks still assess prediction performance, not whether the causal assumptions hold.
What Diagnostics Can and Cannot Establish
A well-calibrated model may still be wrong in ways that matter for causal weighting. For example, grouped calibration can conceal errors within a group, and measured histories can omit a predictor that is important for treatment assignment. Conversely, imperfect calibration in a small subgroup does not automatically mean the whole analysis is unusable; the evidence, sample size, model purpose, and effect on weights all matter.
Likewise, a plausible distribution of fitted probabilities does not establish that all treatment options are sufficiently represented in every relevant history. Nor does it establish that all confounders have been measured or that treatment and censoring are conditionally independent of the relevant potential outcomes. Diagnostics should therefore be read alongside the causal design and assumptions. They are tools for finding model-data incompatibilities and locating their potential consequences, not tests that certify causal validity.
Common Mistake
Check Your Understanding
Use the definitions and diagnostic reasoning from this tutorial to answer the following questions.
- A treatment model predicts a probability of 0.40 for a group of 80 eligible visits, and 28 visits receive treatment. What is the observed treatment proportion, and how does it compare with the mean prediction?
- Why can a model have the correct overall treatment frequency while showing important history-specific prediction errors?
- For a person who receives \(A_t=0\), which fitted treatment probability belongs in the denominator contribution: \(\hat e_t(H_t)\) or \(1-\hat e_t(H_t)\)? Explain.
- What does a very small fitted probability of remaining observed suggest about a censoring-weight contribution for someone who remains observed? Name one reason, other than model misspecification, that could produce such a prediction.
- Why do favorable calibration and residual diagnostics not establish sequential exchangeability or positivity?