Why Time-to-Event Predictions Need Their Own Checks
For a time-to-event pharmacodynamic (PD) endpoint, a model predicts not just whether an event occurs, but also when it occurs. A model can predict a plausible overall event proportion while placing too many events early and too few later. It can also fit an overall event proportion but systematically underpredict events at high exposure. Diagnostics should therefore examine event incidence over time and across exposure levels, rather than relying on a single summary.
The observed data may include right censoring: for some individuals, follow-up ends before an event is observed. A censored individual is known to have remained event-free through their last observed time, but their later event time is unknown. Comparisons must respect this incomplete information. In particular, treating censored individuals as if they never experience the event generally biases a raw event proportion downward.
The hazard is an instantaneous event rate conditional on remaining event-free up to that time; it is not itself an event probability. Exposure may enter the hazard directly or through a time-varying exposure history. For example, a proportional-hazards form may be \(h_i(t)=h_0(t)\exp\{\beta E_i(t)\}\), where \(h_0(t)\) is a baseline hazard, \(E_i(t)\) is exposure, and \(\beta\) describes the modeled exposure association. Diagnostics assess predictive agreement; an observed association between exposure and event risk does not, by itself, establish a causal effect.
Useful comparisons require predictions on the same basis as the observations. For an individual-level prediction, calculate \(1-S_i(t)\) at a prespecified horizon. For a group, average the individual predicted risks, \(\overline{R}(t)=n^{-1}\sum_i[1-S_i(t)]\). This is generally not identical to transforming the average cumulative hazard: averaging and the nonlinear exponential transformation do not generally commute. Predictions should reflect the intended target, such as conditional predictions for the observed individuals or population predictions integrating over individual variability.
Observed Versus Predicted Event Risk
At a clinically meaningful horizon, compare the observed event risk with the model-predicted risk. With complete follow-up through \(t\), the observed risk is the event proportion by \(t\). With right censoring before \(t\), use a censoring-aware estimate, such as the Kaplan–Meier estimate of survival, and calculate observed risk as \(1-\widehat{S}(t)\). This interpretation relies on censoring being independent of event time, at least conditional on variables used in the analysis. If censoring depends on unmodeled event risk, the observed estimate may not be a valid benchmark.
A simple calibration discrepancy for group \(g\) is \(D_g(t)=\widehat{R}_{\mathrm{obs},g}(t)-\overline{R}_{\mathrm{pred},g}(t)\). Positive values mean the observed risk exceeds the predicted risk; negative values mean the model predicts more events than observed. Compare these discrepancies across prespecified exposure strata as well as overall. A systematic increase in \(D_g(t)\) with exposure is a warning of exposure-related hazard miscalibration, though uncertainty and group composition must also be considered.
Worked Example: Comparing Risk Across Exposure Strata
Hypothetical data. At a fixed follow-up horizon, suppose censoring-aware observed risk estimates are 0.0406400920 in a low-exposure stratum and 0.825 in a high-exposure stratum. The corresponding model-predicted risks, averaged over individuals in each stratum, are 0.0203200460 and \(1-\exp(-0.916)\), respectively. The decimal observed risks are possible for Kaplan–Meier estimates and need not be simple event fractions.
For the high-exposure stratum, calculate the predicted risk without first rounding the exponential:
Its observed-minus-predicted discrepancy is
For the low-exposure stratum,
The contrast in discrepancies is
At this horizon, observed risk exceeds predicted risk in both strata, and the excess is larger in the high-exposure stratum. This pattern motivates checking the exposure–hazard relationship, covariate distribution, and uncertainty in the estimates. It does not prove that exposure caused the difference, nor does this contrast alone identify the source of model error.
Check Timing With Simulation
A horizon-specific risk comparison can miss timing errors between horizons. Simulation-based checks address this by generating replicated event-time data under the fitted model and comparing summaries of the observed data with the distribution of those summaries under the model. Useful summaries include numbers of events in time intervals, cumulative event counts at prespecified times, and event-time quantiles when estimable.
The simulation must reproduce the design features that affect the comparison. Keep the observed covariates and exposure histories fixed when assessing predictions conditional on those histories. Simulate event times from the fitted model and, when censoring affects the observed summary, also reproduce the censoring or follow-up process. For population predictions, simulate the relevant between-individual variability as well. A simulation envelope is a reference range generated by the model, not a confidence interval for a parameter and not proof that the model is correct.
Worked Example: Detecting a Timing Error With Interval Counts
Hypothetical data. Suppose 100 individuals have complete follow-up and the fitted model predicts the same constant hazard, \(h(t)=0.20\) per day, for each individual. The survival function is \(S(t)=\exp(-0.20t)\). The observed numbers of events are 24 in days 0–2, 39 in days 2–5, and 37 after day 5.
The model-predicted probability of an event in days 0–2 is \(1-\exp(-0.20\times2)\), so the expected count is
For days 2–5, the probability is \(S(2)-S(5)\), giving
The predicted number of events after day 5 is \(100S(5)=100\exp(-1)=36.7879\ldots\). The predicted cumulative event risk is \(1-\exp(-0.4)=0.3297\) by day 2 and \(1-\exp(-1)=0.6321\) by day 5. The observed proportions are 0.24 by day 2 and 0.63 by day 5. Thus, the total number of events by day 5 is close to the prediction, but substantially fewer events occur in the first interval than expected, with more occurring in the second interval. That pattern is consistent with events occurring later than the model predicts.
To assess whether this discrepancy is unusual, simulate replicated data under the fitted model and compare the interval counts and cumulative counts with their simulated distributions. The three interval counts are dependent because they sum to 100; they should not be treated as three independent tests. A time-specific pattern that repeatedly falls outside appropriate simulation envelopes is more informative than a single isolated discrepancy.
Separate Exposure Miscalibration From Timing Miscalibration
Exposure-related miscalibration and timing error can coexist, but they suggest different questions. If observed risk is consistently higher than predicted in a high-exposure group at several horizons, investigate whether the exposure effect on the hazard is too weak, whether the exposure range is adequately represented, and whether relevant covariates or effect modifiers are missing. If predictions agree at a late horizon but disagree at earlier horizons, investigate the time course of the hazard, the exposure history, and any lag or delay structure.
An exposure-stratified simulation check can compare event counts by both exposure group and time interval. For each stratum, calculate predicted interval counts using \(S_i(a)-S_i(b)\), then generate replicated counts with the same group sizes and exposure patterns. This helps distinguish a broad excess of events in one exposure stratum from a timing shift within that stratum. Use clinically meaningful, preferably prespecified, time horizons and exposure summaries; searching many cut points after seeing the data can produce apparently striking discrepancies by chance.
Diagnostics also depend on what is being predicted. A conditional check that fixes each individual’s estimated random effect can look better than a population-level check that integrates over the random-effects distribution. The former asks whether the model describes individuals given their estimated effects; the latter asks whether it predicts new individuals from the population. State which target is being examined and keep the observed and simulated summaries aligned with it.
Worked Example: Joint Exposure and Timing Check
Hypothetical data. Consider 50 low-exposure and 50 high-exposure individuals, all followed until the event, including beyond day 5. Suppose exposure is constant within each individual, with fitted exponential hazards of 0.10 per day for low exposure and 0.20 per day for high exposure. In the low-exposure group, the observed interval counts are 9, 11, and 30 for days 0–2, days 2–5, and after day 5. In the high-exposure group, they are 12, 22, and 16.
For the low-exposure group, the predicted interval probabilities are \(1-\exp(-0.2)=0.1813\), \(\exp(-0.2)-\exp(-0.5)=0.2122\), and \(\exp(-0.5)=0.6065\). Multiplying by 50 gives expected counts of 9.06, 10.61, and 30.33. The observed counts, 9, 11, and 30, are close to those expectations.
For the high-exposure group, the predicted interval probabilities are \(1-\exp(-0.4)=0.3297\), \(\exp(-0.4)-\exp(-1)=0.3024\), and \(\exp(-1)=0.3679\). Expected counts are therefore
The high-exposure group has 34 observed events by day 5, compared with a predicted count of \(50[1-\exp(-1)]=31.6060\ldots\). More strikingly, only 12 events occur in the first interval, compared with 16.48 expected, while 22 occur in the middle interval, compared with 15.12 expected. The late count is also below its expectation. This pattern suggests a possible timing discrepancy in the high-exposure group along with a modest excess in cumulative risk by day 5. It is not enough to conclude that either discrepancy is statistically unusual: simulate the joint summaries under the fitted model, retaining the two exposure groups and their sizes, and assess the observed pattern against those simulations.
Diagnostics, Alternatives, and Limitations
A practical diagnostic sequence is to define the prediction target and time horizons, compare observed and predicted risks overall and by exposure, inspect event counts across intervals, and then use simulation to judge whether the joint pattern is compatible with the model. The observed-versus-predicted PD plots and calibration concepts used for other endpoint types extend naturally to event risk, but censoring requires a survival estimate or a model-based simulation rather than a naive event proportion.
| Approach | Useful for | Main caution |
|---|---|---|
| Observed-versus-predicted risk at a horizon | Finding overall or exposure-stratified risk discrepancies | One horizon can hide timing errors; censoring must be handled |
| Kaplan–Meier observed risk by horizon | Estimating observed event risk with right-censored data | Requires suitable independent-censoring assumptions and has sampling uncertainty |
| Simulation-based event-time checks | Checking event counts, timing, and joint patterns under the fitted model | Results depend on a correctly specified simulation and censoring scheme |
| Likelihood-based residual diagnostics | Examining individual contributions and unusual observations | May be difficult to interpret directly as calibration over time or exposure |
Common warning signs include discrepancies that persist across adjacent horizons, systematic residual patterns by exposure, observed interval counts outside simulation envelopes, and poor agreement in one group despite acceptable overall calibration. Sparse events, heavy censoring, small exposure strata, and few individuals at risk late in follow-up reduce diagnostic resolution. Apparent late-time disagreement may be driven by very few remaining individuals, so always report the risk set and uncertainty alongside the curve or summary.
Common mistake: “The predicted event risk by day 5 matches the observed risk, so the model predicts event times correctly.” This is not justified: different event-time distributions can have the same risk at day 5. Compare risks at multiple times and examine interval counts or simulated event-time summaries. Another mistake is calculating events divided by the original sample size when some individuals were censored before the horizon; those individuals’ event status by that horizon is unknown, not event-free.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A model predicts cumulative hazard \(H(4)=0.30\). What is its predicted event risk by time 4?
- At a chosen horizon, observed risk is 0.40 and predicted risk is 0.32. Interpret the discrepancy using the observed-minus-predicted convention.
- Why can a model match observed risk at day 10 but still predict event timing poorly?
- Why is the raw fraction of observed events generally not a suitable event-risk estimate when some individuals are censored before the horizon?
- In a simulation-based check, what should be preserved or simulated if the diagnostic is intended to assess predictions conditional on observed exposure histories?
Solutions. (1) The predicted risk is \(1-\exp(-0.30)=0.2592\), or about 25.9%. (2) The discrepancy is \(0.40-0.32=0.08\), so observed risk is 8 percentage points higher than predicted at that horizon. (3) The event-time distributions may differ before day 10 while reaching similar cumulative risk by day 10; compare multiple horizons and interval counts. (4) Those censored before the horizon have unknown event status at that horizon, so treating them as event-free can bias the fraction downward; use a censoring-aware estimate under appropriate censoring assumptions. (5) Keep the observed exposure histories and relevant covariates fixed, simulate event times under the fitted model, and reproduce censoring or follow-up when needed for the observed summary.