Checking Multi-State Predictions Against Observed Occupancy

What Does It Mean to Check a Multi-State Prediction?

A multi-state prediction assigns probabilities to the possible states a person may occupy at a specified time. Checking such predictions means comparing them with estimates of occupancy observed in follow-up data. The comparison is useful only when the predictions and observed estimates refer to the same states, time origin, time horizon, and target population. A mismatch can reflect model miscalibration, a mismatch in those targets, or limitations in the observed-data estimate.

Consider an illness-death model with state 0 for alive and not ill, state 1 for alive and ill, and state 2 for dead. Death is absorbing: once a person enters state 2, that person remains there. At time \(t\), the three state-occupation probabilities for a target group are \(P(X(t)=0)\), \(P(X(t)=1)\), and \(P(X(t)=2)\), where \(X(t)\) denotes state at time \(t\). They are nonnegative and sum to one.

Definition: A state-occupation prediction at time \(t\) is a vector of probabilities, one for each state, describing the predicted distribution of state at \(t\) for a specified starting state or starting distribution and target population. An observed occupancy estimate estimates that same distribution from follow-up data. A comparison is well aligned only if both quantities use the same state definitions, time origin, time point, and population target.

A covariate-specific prediction is conditional on a particular covariate profile. An ordinary pooled Aalen-Johansen estimate, by contrast, is an estimate for the cohort represented by its pooled risk sets, subject to its censoring assumptions. Those are not automatically the same target. A model prediction for a particular profile should not be judged against a pooled estimate as if the latter were the observed outcome for that profile.

Assumptions and a Direct Aalen-Johansen Check

For an ordinary pooled Aalen-Johansen (AJ) estimate, censoring must be independent of the future transition process marginally in the population being analyzed. If censoring is independent only conditional on covariates, a pooled AJ estimate does not automatically estimate marginal occupancy for the original cohort: changing covariate composition in the risk sets can change the pooled transition estimates. With covariate-dependent censoring, use an approach that accounts for it, such as suitable stratum-specific estimates followed by standardization, or another appropriate adjusted method.

The AJ estimator combines observed transition counts with the numbers at risk in each state. In a model with several possible transitions, it estimates a transition-probability matrix by multiplying the estimated transition matrices across event times. Applying that product to the starting-state distribution gives estimated occupancy. The small calculation below shows the update at two event times; it is not a substitute for checking all event times in a full analysis.

Worked Example: Updating Illness-Death Occupancy

Suppose, hypothetically, 100 people start in state 0. At the first event time, 100 are at risk in state 0; 10 move from 0 to 1 and 5 move from 0 to 2. No one is yet at risk for a 1-to-2 transition at that instant. The estimated transition probabilities from state 0 are therefore \(10/100=0.10\) to state 1, \(5/100=0.05\) to state 2, and \(85/100=0.85\) remaining in state 0. The occupancy vector after this update is \((0.85,0.10,0.05)\).

At a later event time, suppose 80 are at risk in state 0, with 8 transitions to state 1 and 4 to state 2; 9 are at risk in state 1, with 3 transitions to state 2. The transition rows at this time are:

P_2 = \begin{pmatrix} 68/80 & 8/80 & 4/80 \\ 0 & 6/9 & 3/9 \\ 0 & 0 & 1 \end{pmatrix} = \begin{pmatrix} 0.85 & 0.10 & 0.05 \\ 0 & 0.6667 & 0.3333 \\ 0 & 0 & 1 \end{pmatrix}.

The updated occupancy is the previous row vector multiplied by this transition matrix:

(0.85,0.10,0.05)P_2 = (0.7225,\;0.1517,\;0.1258).

For example, occupancy in state 1 is \(0.85(0.10)+0.10(6/9)=0.1517\), to four significant figures. The three entries sum to 1, apart from rounding. This calculation also shows why occupancy is not just the proportion with an observed transition of a particular type: people can enter state 1 at one time and later move to death.

This product calculation is appropriate when the transition model and time scale support the transition probabilities being used. For model-based predictions from transition-specific hazard models, use predictions constructed under the model’s stated assumptions. For example, a Markov prediction uses current state and the model’s specified time scale; if transition rates depend on time spent in the current state, a model that ignores that duration may not be appropriate. The comparison itself cannot repair a misspecified prediction model.

Aligning a Covariate-Specific Prediction with Observed Occupancy

There are two common comparisons. First, for a covariate-defined subgroup, compare the model prediction for that subgroup with an observed estimate calculated in that subgroup, provided the subgroup has adequate data and the estimator’s assumptions are credible. Second, for a marginal target cohort, average individual model predictions over that cohort’s covariate distribution and compare the result with an observed estimate targeting the same cohort. The averaging must use the intended target distribution, not an arbitrary risk-set distribution at each time.

When conditional predictions are available for covariate profiles \(z_i\), a simple empirical standardization over \(n\) people uses the average predicted probability for each state:

\widehat{p}^{\,\mathrm{pred}}_j(t) = \frac{1}{n}\sum_{i=1}^{n} \widehat{P}\{X(t)=j\mid z_i,\ X(0)=0\}, \qquad j=0,1,2.

This creates predictions for the chosen covariate distribution. The observed comparator must target that same distribution. If the observed AJ estimate is pooled and censoring depends on covariates, the marginal-independence condition for ordinary pooled AJ may fail even when censoring is independent within covariate strata. In that setting, use a suitable adjusted observed estimate rather than treating the pooled estimate as a valid marginal benchmark.

Worked Example: Comparing a Standardized Prediction with AJ Occupancy

Suppose, hypothetically, a fitted model gives a 12-month prediction for a target cohort of \((0.78,0.12,0.10)\) for states 0, 1, and 2 after averaging its individual predictions over the cohort’s baseline covariates. An observed estimate that targets the same cohort is \((0.80,0.11,0.09)\). Define the discrepancy in state \(j\) as predicted minus observed occupancy:

d_j(12)=\widehat{p}^{\,\mathrm{pred}}_j(12)-\widehat{p}^{\,\mathrm{obs}}_j(12).

The state-specific discrepancies are \(d_0=0.78-0.80=-0.02\), \(d_1=0.12-0.11=0.01\), and \(d_2=0.10-0.09=0.01\). Thus the model predicts 2 percentage points less occupancy in state 0 and 1 percentage point more in each of states 1 and 2 at 12 months. The discrepancies sum to zero because both probability vectors sum to one. They describe a difference between estimates, not proof that the model is wrong or that a particular transition caused the difference.

Before interpreting this as calibration evidence, verify how the observed estimate was formed. If censoring is plausibly independent only within covariate strata, a pooled AJ estimate is not necessarily a suitable comparator for this marginal prediction. An adjusted or standardized observed estimate must be used, with its target population stated.

Plots and Risk-Set Checks

A useful diagnostic plot shows predicted and observed occupancy over time, separately for each state. Plot predictions as smooth curves or connected points at the evaluation times and observed estimates as step functions or points, with uncertainty intervals when available. A plot of state-specific differences can make the direction and timing of departures easier to see. Since the state probabilities sum to one, discrepancies across states are related: an excess in one or more states must be balanced by deficits elsewhere.

Worked Example: Reading a State-Occupation Calibration Plot

Suppose, hypothetically, a standardized model prediction and a properly adjusted observed estimate are compared at 6, 12, and 18 months:

Time	Predicted state 0	Observed state 0	Predicted state 1	Observed state 1	Predicted state 2	Observed state 2
6 months	0.82	0.84	0.13	0.11	0.05	0.05
12 months	0.68	0.70	0.20	0.18	0.12	0.12
18 months	0.55	0.60	0.20	0.17	0.25	0.23

At 18 months, the state 0 difference is \(0.55-0.60=-0.05\), or 5 percentage points lower predicted occupancy. The state 1 difference is \(0.20-0.17=0.03\), and the state 2 difference is \(0.25-0.23=0.02\). These differences sum to zero. If a plot showed this growing late-time pattern, it would warrant investigation rather than a conclusion based on one point alone. Possible explanations include transition-model misspecification, a changing target population, few people remaining under observation, or censoring that makes the observed estimator unreliable.

At each selected time, inspect the number at risk in every state, the number of observed transitions along each possible edge, and the number censored while in each state. The uncertainty of late occupancy estimates can become substantial when risk sets are small, even if the plotted point estimates look smooth. Also check whether the prediction is extrapolating beyond times with adequate follow-up or beyond covariate profiles represented in the data.

A risk-set table should use the same time points and state definitions as the plot. In an illness-death model, a total cohort size alone is not enough: the transitions 0-to-1, 0-to-2, and 1-to-2 draw information from different state-specific risk sets. Sparse risk sets for one transition can produce unstable estimates in the states reached through that transition. A sharp change in discrepancy near a time with very few people at risk may be weak evidence of systematic miscalibration.

Short Justification and Practical Workflow

The need for target alignment follows directly from what an occupancy probability averages. For a marginal target population with covariate distribution \(F_Z\), state \(j\) occupancy is the average conditional probability across that distribution:

P\{X(t)=j\} = \int P\{X(t)=j\mid Z=z\}\,dF_Z(z).

A prediction for one value \(z_0\) estimates the integrand at \(z_0\), not the integral. Likewise, a pooled observed estimate affected by changing covariate composition in its risk sets need not estimate the integral for the original cohort. Thus a discrepancy can arise even if the conditional model is well calibrated, simply because the two quantities average over different populations. Standardizing the model predictions and choosing a compatible observed estimator address this mismatch; they do not guarantee that either is unbiased under violated assumptions.

Fix the estimand.
Specify the starting state or distribution, target population, state definitions, and evaluation times.

Construct comparable quantities.
Obtain predictions for the specified target, and select an observed estimator whose population and censoring conditions match it.

Plot and tabulate.
Compare each state’s occupancy over time and report pointwise discrepancies at prespecified times, alongside uncertainty and risk-set counts.

Investigate departures.
Check transition-specific event counts, censoring patterns, covariate representation, and the prediction model’s assumptions before attributing a difference to miscalibration.

Uncertainty, Alternatives, and Limitations

Observed occupancy estimates and model-based predictions both have sampling uncertainty. A difference between two point estimates is not, by itself, a formal test of model fit. If uncertainty in the difference is needed, a bootstrap can resample people, refit the prediction model, and recompute the observed estimates and predictions in each resample. The resampling scheme should respect the design, such as clustering when observations are dependent within clusters. This captures uncertainty in both parts of the comparison when the full procedure is repeated.

Pointwise intervals at separate times do not provide simultaneous coverage over an entire curve. Nor does overlap of two confidence intervals establish that the predictions and observations agree. A validation sample separate from the model-fitting sample can provide a more honest assessment of predictive performance than apparent agreement in the data used to fit the model. Even in validation data, a limited sample or short follow-up can make later-time comparisons imprecise.

The main alternatives answer related but different questions. A transition-specific diagnostic examines whether particular modeled transitions agree with their observed data; it may help localize a problem but is not itself an occupancy comparison. A pooled AJ curve is a natural nonparametric benchmark when its marginal censoring assumption and target are appropriate. Stratified AJ estimates can address covariate-dependent censoring when censoring is independent within strata, but small strata may be unstable; standardizing across strata also requires a clear target distribution. Other adjusted methods may be needed when those conditions are not adequate. No single plot or benchmark proves that a multi-state model is correctly specified.

Common Mistakes and Failure Modes

Comparing a profile prediction with a pooled estimate. The prediction is conditional on a covariate profile, while the pooled estimate represents a population average. Compare like with like, or explicitly describe the difference in targets.
Assuming conditional independent censoring is enough for pooled AJ. If censoring is independent only conditional on covariates, pooled risk sets can change their covariate composition over time. Use an adjusted or appropriately stratified and standardized observed estimate.
Ignoring state-specific risk sets. A total sample size can conceal a nearly empty risk set for a particular transition. Report numbers at risk by state and inspect transition counts.
Calling a small pointwise difference proof of calibration. Sampling variability and model-fitting uncertainty matter. Show uncertainty and avoid treating overlap of intervals as a definitive test.
Interpreting occupancy differences as transition effects. Occupancy combines the possible paths through the state system. A difference in death occupancy, for example, does not identify which transition model produced it.

Key takeaway: A meaningful check compares predicted and observed state occupancy for the same population, states, and times. For ordinary pooled AJ, marginal independence of censoring is required; conditional independence alone does not establish that the pooled estimate is a valid marginal benchmark.

Check Your Understanding

Answer each question, then check the concise solutions below.

A model predicts state 0 occupancy of 0.70 for a particular covariate profile. Is a pooled AJ estimate of 0.74 automatically the appropriate comparator? Why or why not?
At one time point, predicted occupancy is \((0.60,0.25,0.15)\) and observed occupancy is \((0.63,0.22,0.15)\). Calculate the prediction-minus-observation differences.
Why might ordinary pooled AJ fail to estimate marginal occupancy when censoring is independent only conditional on a covariate?
At late follow-up, a plot shows disagreement and only two people remain at risk in state 1. What should be checked before interpreting the difference as evidence of model failure?
Why do discrepancies across all three states sum to zero when both occupancy vectors are valid probability distributions?

Solutions. (1) No. The prediction is conditional on a profile, whereas pooled AJ is a cohort-level estimate; align the target, for example with a suitable subgroup comparison or standardized prediction. Also verify the censoring assumptions for the observed estimate. (2) The differences are \(0.60-0.63=-0.03\), \(0.25-0.22=0.03\), and \(0.15-0.15=0\). (3) Covariate-dependent censoring can change the covariate mix of pooled risk sets, so pooled transition estimates need not represent the original cohort’s marginal occupancy. (4) Check state-specific event counts and censoring, the time horizon and target alignment, and the uncertainty; two people at risk provide very little late-time information. (5) Each vector sums to one, so \(\sum_j(\widehat{p}^{\,\mathrm{pred}}_j-\widehat{p}^{\,\mathrm{obs}}_j)=1-1=0\).