From Model Fit to Event-Time Prediction
A fitted event-time model can describe how exposure relates to hazard and still predict event experience poorly. Calibration checks address a different question from checking individual residuals: do the model’s predicted event probabilities agree with the outcomes observed in the population and at the times that matter? For PK/PD applications, this comparison should be made across relevant exposure groups, because predictions may be adequate at low exposure but systematically wrong at high exposure, or vice versa.
A survival comparison concerns remaining free of the event through time \(t\). For competing risks, a cause-specific cumulative-incidence comparison concerns having experienced a particular event type by time \(t\), before another event type occurs. The observed and predicted quantities must use the same time origin, event definition, exposure grouping, and target population. A curve comparison is not meaningful if, for example, the observed curve describes the people actually in a group but the predicted curve describes a differently standardized population.
Make the Observed and Predicted Quantities Comparable
For a single event type, the Kaplan–Meier estimate provides an observed estimate of survival when censoring is independent of event time, either marginally within the comparison group or conditional on variables used to account for censoring. At each observed event time \(t_j\), let \(n_j\) be the number at risk immediately before that time and \(d_j\) the number of events. The estimate is the product of the conditional probabilities of surviving each event time:
If the model predicts survival \(S_i(t)\) for individual \(i\), a group-level prediction for a defined set of \(N_g\) individuals is often the average \(\bar S_g(t)=N_g^{-1}\sum_{i\in g}S_i(t)\). This is a marginal prediction for that group, not the prediction for a hypothetical person with average covariates. The latter generally differs in nonlinear models. If the target is a standardized population, average the individual predictions over that same target population for every exposure group.
For competing risks, the observed cumulative incidence is commonly estimated with the Aalen–Johansen estimator. At each event time, the estimated probability of still being free of any event is multiplied by the observed cause-specific fraction of events among those at risk. This accounts for the fact that a competing event prevents subsequent occurrence of the event of interest. Thus, one minus a cause-specific hazard-based survival curve is not, in general, the observed cumulative incidence for that cause.
Worked Examples of Curve Calibration
Worked Example: Comparing Survival at a Fixed Time
Suppose a hypothetical low-exposure group has 10 participants. Two events occur at month 2, one participant is censored at month 3, and two events occur at month 4. Assume censoring is independent of event time within this group, and there is no delayed entry. Calculate the Kaplan–Meier estimate at month 4 and compare it with a model-predicted group survival of 0.60.
At month 2, 10 participants are at risk and 2 have events. The survival factor is \(1-2/10=0.80\). After those events, 8 remain at risk. The censoring at month 3 reduces the risk set to 7, but does not create a survival factor. At month 4, 2 of those 7 have events, giving a factor of \(1-2/7=5/7\). Therefore,
The observed estimate is 0.5714 and the model prediction is 0.60, a difference of \(-0.0286\), or about 2.86 percentage points. This small example illustrates the calculation, not strong evidence of a calibration problem: with only 10 participants, sampling uncertainty is substantial. The independent-censoring assumption also matters; if censoring preferentially removes participants with high event risk, the Kaplan–Meier estimate may not represent the intended group.
Worked Example: Checking Competing-Risk Predictions
In a hypothetical group of 20 participants, by time 1 there are 2 events of cause 1 and 1 event of cause 2, with no censoring. By time 2, among the 17 still event-free, there are 3 cause-1 events and 2 cause-2 events. Calculate the observed cumulative incidences and compare them with model predictions at time 2 of 0.22 for cause 1, 0.17 for cause 2, and survival of 0.61.
At time 1, estimated survival falls from 1 to \(17/20=0.85\). The cause-1 cumulative-incidence increment is \(1\times 2/20=0.10\); the cause-2 increment is \(1\times 1/20=0.05\). At time 2, the survival immediately before events is 0.85. The increments are \(0.85\times 3/17=0.15\) for cause 1 and \(0.85\times 2/17=0.10\) for cause 2. Thus,
The model predictions sum to \(0.22+0.17+0.61=1.00\), as they should for the two event types and remaining event-free status at this time. The observed values also sum to \(0.25+0.15+0.60=1.00\). The prediction is 0.03 lower than observed for cause 1, 0.02 higher for cause 2, and 0.01 higher for survival. The small differences alone do not establish whether the model is adequate; uncertainty and patterns across time and exposure groups must also be considered.
Simulation-Based Predictive Checks
A curve can look close to the observed estimate by chance, and a fitted model’s predictions are themselves uncertain. Simulation-based predictive checks make that uncertainty visible. A check begins by specifying a discrepancy that would reveal a relevant failure: for example, the number of events by a clinically relevant time, survival at a set of times, cause-specific cumulative incidence, or the distribution of event times within an exposure group.
The model is then used to generate replicated event outcomes under the same design and analysis conditions as the observed data. Replications should preserve exposure and relevant covariates, and should include the event process, competing event types where applicable, and censoring or delayed entry in a way consistent with the data-generating assumptions. For each replication, calculate the same discrepancy or curve summary as for the observed data. Comparing the observed value with the distribution across replications shows whether the observed feature is typical under the fitted model.
Specify the time range, exposure groups, event type, and summary to check. Select summaries that can expose meaningful errors, not only an overall average.
Simulate event times and types from the fitted model, including relevant individual covariates or random effects. Apply the observation process, including censoring, when reproducing the observed-data summaries.
For every replication, calculate the same event counts, observed curves, or other chosen statistics used for the actual data.
Plot or tabulate the observed summary against its simulated distribution. Look for consistent deviations by time, exposure group, or event type rather than interpreting one isolated discrepancy.
One short justification explains why this comparison is informative. If the model and observation process adequately describe the data, then a new dataset generated under those assumptions should produce summaries with the same general distribution as the observed summary. Therefore, an observed statistic far into a simulated distribution’s tail signals a feature that the fitted model rarely reproduces. It does not identify the cause of the failure by itself.
Worked Example: A Predictive Check for Event Counts
Suppose a hypothetical fitted model predicts the number of events by month 6 in a high-exposure group. The observed count is 12. In 200 simulated datasets, the mean count is 16, and 9 simulations have 12 or fewer events. Use the simulations to describe the discrepancy.
The observed count is 4 events below the simulated mean. Its lower-tail simulation proportion is \(9/200=0.045\). With the common finite-simulation adjustment, the estimated lower-tail proportion is \((9+1)/(200+1)=0.0498\). Thus, about 5% of these simulations produced 12 or fewer events. The observation is near the lower tail of the predictive distribution, suggesting that the model may predict too many events by month 6 in this group, or equivalently too little event-free survival.
This proportion is a simulation-based diagnostic, not the probability that the model is true or false. It is also not automatically a formal hypothesis-test p-value: the result depends on the chosen summary, fitted parameters, and simulation procedure. The check should be repeated for other times and groups. If the simulation omitted censoring that affects the observed event count, the comparison would not reproduce the data collection process and could be misleading.
Diagnostics, Alternatives, and Limitations
A useful diagnostic is not simply whether two curves cross or whether one point differs. Look for structured discrepancies: predictions that are systematically too optimistic early and too pessimistic later, an exposure gradient in the observed curves that the model fails to reproduce, or a persistent mismatch for one competing event type. Plotting observed estimates and predicted curves on the same axes by exposure group makes these patterns easier to see. Simulated envelopes can show the variation expected under the model, but their interpretation depends on how the simulations were constructed.
There are two common simulation targets. A conditional check holds fitted parameter estimates fixed and asks whether new outcomes generated at those estimates reproduce the observed patterns. A parameter-uncertainty check also draws or otherwise accounts for uncertainty in the fitted parameters before generating outcomes. A parametric bootstrap can additionally simulate and refit datasets, which assesses the behavior of the whole fitting procedure. These checks answer related but different questions: holding estimates fixed may understate uncertainty, while refitting is more computationally demanding and can reveal instability in estimation.
Direct nonparametric curve comparisons and simulations complement rather than replace one another. Kaplan–Meier or Aalen–Johansen curves show what is observed with relatively few model-shape assumptions, subject to censoring assumptions and limited precision in small groups. Model-based predictions can account for covariates and support standardization, but depend on the specified model. A single calibration statistic is compact but can hide when and where predictions fail. Individual residual diagnostics, including the earlier material on Cox–Snell and martingale residuals, can reveal other forms of misspecification; they do not substitute for checking group-level predicted probabilities.
Several limitations deserve attention. Heavy censoring makes late-time observed curves unstable because few participants remain at risk. Small exposure groups produce noisy estimates, so apparent separation may be sampling variation. Grouping a continuous exposure can conceal a nonlinear calibration pattern within a group. Conversely, a comparison of group averages can conceal poor predictions for important covariate subgroups. Finally, simulation checks can only detect discrepancies visible to the chosen summaries and under the assumed simulation design. They cannot guarantee that the model will predict well in a new population or under exposure levels not represented in the data.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a single-event analysis, what does \(S(t)\) represent, and why must observed and predicted survival use the same target population?
- In a competing-risks analysis, why is the cumulative incidence of cause 1 not generally equal to one minus survival calculated from the cause-1 hazard alone?
- A model’s observed event count is 8, while 100 simulations produce 6 counts at or below 8. What is the unadjusted lower-tail simulation proportion, and what does it indicate?
- Name two ways an apparently good overall curve comparison can conceal prediction failure.
Solutions. (1) \(S(t)=P(T>t)\), the probability of remaining event-free beyond \(t\). Matching the target population ensures the observed and predicted probabilities refer to the same individuals or covariate distribution. (2) A competing event prevents later occurrence of cause 1. Its probability must be included in the event-free survival that contributes to cause-1 incidence; treating cause 1 as if other events could not occur misstates that probability. (3) The proportion is \(6/100=0.06\), or 6%. The observed count is near the lower tail of these simulations, so it may indicate fewer observed events than the model typically predicts; it is a diagnostic, not automatically a formal p-value. (4) Examples include opposing errors at early and late times that cancel in an overall summary, and poor calibration within covariate subgroups hidden by a group average. Small groups or heavy censoring can also make a visually close comparison uninformative.