Why Plot Prediction Error Across Follow-Up?
A validation result at one horizon can hide an important pattern. A model may predict a transition well at six months, systematically overpredict it at twelve months, and underpredict it later. A time-indexed plot makes that change visible: it places observed and predicted transition probabilities on the same time scale and shows their difference at each horizon. The aim is not merely to produce another summary score, but to locate when agreement changes and for which transition.
The plot is meaningful only after specifying the prediction origin, the target population, the state or transition of interest, and how observed probabilities are estimated. For example, a probability of occupying state 1 at time \(t\) after starting in state 0 is not necessarily the probability of ever entering state 1: a person could enter state 1 and later move elsewhere. In contrast, the cumulative incidence of an absorbing destination is nondecreasing over time.
The target in this definition is marginal over the validation population's covariate distribution: it compares the average of individual predictions with the population's observed probability. It does not compare a prediction evaluated at mean covariates with the observed outcome. In nonlinear models, those two predictions need not be equal. A fixed prediction origin and a clearly defined validation cohort are also essential: changing the cohort or the origin across horizons can make apparent time trends reflect changing composition rather than changing performance.
Assumptions and Construction
With complete follow-up to every plotted horizon, the observed proportion in state \(j\) among those who began in state \(i\) can estimate the target probability. In practice, follow-up is often censored. Then the observed curve must come from an estimator that accounts for censoring and matches the estimand. For example, an Aalen–Johansen estimate may be appropriate for state-occupation probabilities under the relevant multi-state assumptions; another censoring-adjusted estimator may be needed for a different target. Simply taking the proportion with a recorded state at each horizon can be biased if people lost to follow-up differ in their future outcomes.
The validation cohort should represent the population to which the predictions are meant to apply, and predictions and observed estimates must refer to the same time origin, eligibility criteria, state definitions, and covariate information. The observed-probability estimator must be justified under the data's censoring and event-history structure. For example, independent censoring, possibly conditional on measured information, is needed for standard censoring adjustments to identify the target. If a model or estimator requires a Markov assumption, that assumption must be plausible for the target being estimated; it should not be presumed simply because the data have been arranged as a multi-state process.
At each prespecified horizon, calculate the mean predicted probability over the same target cohort and obtain the observed estimate. Display both curves against time, or display the signed error curve directly. Showing both is often useful: the first view makes absolute risk levels easy to compare, while the second gives a direct visual measure of discrepancy. Use a common probability scale from zero to one for state probabilities, and label the origin, destination, time units, and population in the plot.
Specify the initial state, destination or state-occupation outcome, prediction origin, target cohort, and time horizons.
Average subject-specific predictions at each horizon and estimate observed probabilities using a method appropriate to the outcome and censoring.
Plot both curves and, when useful, plot \(\hat e_{ij}(s,t)\). Keep the sign convention explicit.
Show uncertainty for the observed estimates or errors, then judge whether a feature is sustained and practically important rather than reacting to one noisy horizon.
Why the Difference Is a Useful Diagnostic
The signed difference has a direct interpretation because the two terms estimate the same marginal probability. If the estimates are unbiased for their respective targets, then at a fixed horizon the error estimates the difference between the observed target probability and the model's mean predicted probability. The calculation follows by subtracting the average prediction from the observed probability:
Thus, if the observed probability is 0.30 and the mean prediction is 0.34, the error is \(-0.04\), or four percentage points of overprediction. Plotting this quantity at successive horizons shows whether the discrepancy grows, shrinks, or changes sign. This is a marginal calibration diagnostic: it does not prove that predictions are calibrated for every individual or every risk subgroup. Overprediction for some people can cancel underprediction for others.
Worked Example: A Growing Error for Illness-State Occupancy
Suppose a hypothetical validation cohort has 200 people who begin in state 0, an illness-free state, at time \(s=0\). The target is the probability of occupying illness state 1 at each horizon, regardless of any later history. Assume follow-up is complete at these horizons, so the observed proportions are directly available. The model's predictions have already been calculated for each person.
| Horizon (months) | Observed in state 1 | Observed probability | Mean predicted probability |
|---|---|---|---|
| 6 | 36 of 200 | 0.18 | 0.20 |
| 12 | 62 of 200 | 0.31 | 0.29 |
| 24 | 86 of 200 | 0.43 | 0.38 |
At six months, the observed probability is \(36/200=0.18\), so
The model overpredicts by 0.02, or two percentage points. At twelve months,
so the direction has reversed: the mean prediction is now two percentage points below the observed estimate. At 24 months,
The model underpredicts illness-state occupancy by five percentage points at that horizon. A plot of both curves would show their ordering crossing between six and twelve months; the error plot would show a move from negative to positive values and then a larger positive discrepancy. These figures describe state occupancy, which need not increase monotonically if movement out of state 1 is possible. The example's direct observed proportions are appropriate only because follow-up is complete at the stated horizons.
Uncertainty, Time Patterns, and Plot Choices
An observed probability estimate varies across validation samples. Its uncertainty is often greater at later horizons, when fewer people remain under observation or fewer relevant transitions have occurred. A time-indexed plot should make that limitation visible, for example with pointwise confidence intervals for observed probabilities or uncertainty intervals for the error. If predictions are treated as fixed conditional on the fitted model, uncertainty in the error still includes uncertainty in the observed estimate; if the aim is to reflect model-fitting variability too, resampling or other procedures must repeat the relevant model-fitting and validation steps.
Estimates at different horizons are correlated: they are based on overlapping event histories and often the same subjects. Pointwise intervals describe uncertainty at individual horizons, not simultaneous coverage of the whole curve. A handful of intervals or small departures from zero should therefore not be read as a formal test of a time-varying pattern. When simultaneous uncertainty is needed, use a method designed to cover the curve jointly. For estimating sampling variability, subject-level resampling should preserve each person's complete event history rather than resampling their records independently.
Choose the time grid before interpreting the plot when possible. A very dense grid can make random fluctuations look like meaningful oscillation, while a very sparse grid can miss an important reversal. A smooth curve can aid visual reading, but smoothing should not obscure the underlying estimates or create apparent detail unsupported by the data. Show the horizon-specific estimates, or make the smoothing approach and its purpose clear.
The main alternatives answer different questions. A single-horizon calibration summary can be compact but cannot reveal when error develops. Calibration curves across predicted-risk groups address how observed risk varies across prediction levels at a selected horizon; they are not a substitute for following the marginal error across time. Residual diagnostics can reveal subject-level or transition-specific patterns, while the plot here focuses on population-average agreement. A single overall score can summarize performance but may conceal errors of opposite signs at different times or destinations.
Worked Example: Interpreting Censoring-Adjusted Estimates
Consider a hypothetical cohort of 150 people starting in state 0. Some are censored before the later horizons, so the recorded proportion in the destination among those still observed is not automatically a valid estimate of the full-cohort probability. Suppose an appropriate censoring-adjusted estimator, justified for this setting, gives the observed estimates below. The table also gives the mean predictions for the same target cohort.
| Horizon (months) | Observed estimate | Mean prediction | Error |
|---|---|---|---|
| 6 | 0.12 | 0.10 | 0.12 − 0.10 = 0.02 |
| 12 | 0.25 | 0.22 | 0.25 − 0.22 = 0.03 |
| 18 | 0.30 | 0.34 | 0.30 − 0.34 = −0.04 |
The positive errors at six and twelve months indicate underprediction by two and three percentage points, respectively. The error is negative at 18 months, indicating overprediction by four percentage points. If an illustrative 95% confidence interval for the observed estimate at 12 months were 0.19 to 0.31, the point prediction of 0.22 would lie inside it. That overlap alone does not establish good calibration: the interval is for the observed probability, not a direct test of the prediction error, and its width may allow materially different values. A plot should present the interval with its target clearly identified and should avoid treating one horizon as decisive evidence of the whole time pattern.
Competing Destinations and Diagnostic Pitfalls
When a starting state has multiple possible destinations, inspect destination-specific errors rather than relying only on a combined total. Opposite errors can cancel: a model may give the total probability of leaving the starting state correctly while misallocating probability between destinations. At each horizon, predictions must also remain coherent: probabilities for mutually exclusive states should be nonnegative and sum to one, subject to the state structure and the event definition. Observed estimates should be interpreted on the same scale.
Worked Example: Opposing Errors Across Two Destinations
In a hypothetical cohort starting in state 0, suppose destinations A and B are absorbing competing outcomes. The table gives observed cumulative incidence estimates and mean predicted probabilities at three horizons. For this example, assume complete ascertainment and that the destination definitions are mutually exclusive.
| Horizon | Observed A | Predicted A | Error A | Observed B | Predicted B | Error B |
|---|---|---|---|---|---|---|
| 6 months | 0.15 | 0.12 | +0.03 | 0.06 | 0.08 | −0.02 |
| 12 months | 0.23 | 0.20 | +0.03 | 0.11 | 0.14 | −0.03 |
| 24 months | 0.25 | 0.29 | −0.04 | 0.28 | 0.24 | +0.04 |
At 12 months, the errors are \(0.23-0.20=0.03\) for A and \(0.11-0.14=-0.03\) for B. Their sum is zero, so the predicted total incidence in A or B is \(0.20+0.14=0.34\), exactly the observed total \(0.23+0.11=0.34\). Yet the model underpredicts A and overpredicts B by three percentage points each. At 24 months, the errors again cancel: \(-0.04+0.04=0\). A plot of only the combined event probability would conceal both destination-specific discrepancies. Plotting each destination's curves or errors separately reveals them.
The predicted totals in this example are 0.20, 0.34, and 0.53 at six, twelve, and 24 months. They are each below one, as required for coherent competing cumulative incidences. The observed totals are 0.21, 0.34, and 0.53. Since the destinations are absorbing, each destination's cumulative incidence should also be nondecreasing; the example's observed and predicted series satisfy that property.
A frequent mistake is to conclude that a model is well calibrated because the observed and predicted curves are close at the final horizon. The wrong reasoning is that agreement at the endpoint implies agreement throughout follow-up. The correction is to inspect intermediate horizons and the signed error curve: early overprediction and later underprediction can disappear in a single endpoint summary. The reverse mistake is to overinterpret one noisy crossing as a genuine change in model performance. Check the uncertainty, number of people still contributing information, and whether the pattern persists across nearby horizons before drawing a substantive conclusion.
Other failure signals include observed curves that are erratic where few events support estimation, probabilities outside the valid range, nonmonotone cumulative incidences for absorbing destinations, or competing-state probabilities whose sums are incoherent. Such features can indicate sparse information, a mismatched estimator, coding errors, or predictions that do not respect the state structure. They are reasons to investigate, not automatic proof of a particular cause. Finally, a marginal time plot cannot show which individuals are mispredicted or whether errors differ across subgroups. Use complementary diagnostics when those questions matter.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- At 12 months, the observed estimate is 0.27 and the mean predicted probability is 0.31. Calculate the signed error and interpret it.
- Why can the proportion observed among uncensored people at a horizon fail to estimate the target probability?
- For two competing destinations, the observed errors are +0.02 and −0.02. What can and cannot be concluded from their sum?
- A state-occupation probability decreases between two horizons. Is that necessarily an error in the plot? Explain.
- What does it mean if the observed and predicted curves meet at the final horizon but differ earlier?
Solutions. (1) \(\hat e=0.27-0.31=-0.04\). The model's mean prediction exceeds the observed estimate by four percentage points. (2) Censoring may be related to future outcomes, so the people still observed may not represent the original target cohort; a suitable censoring-adjusted estimator and its assumptions are needed. (3) The combined error is zero, so the sum of the two estimated destination probabilities agrees at that horizon. It does not establish calibration for either destination separately; the equal and opposite errors remain. (4) No. Occupancy can decrease if people move out of the state. A decrease would be suspect for the cumulative incidence of an absorbing destination, which should be nondecreasing. (5) Endpoint agreement does not imply agreement over follow-up. Inspect intermediate horizons and the uncertainty to determine whether the earlier discrepancy is meaningful.