Observed-to-Expected Event Ratios

Why Compare Observed and Expected Events?

A joint model can use a participant’s longitudinal history to predict the probability of an event by a chosen prediction horizon. In a validation cohort, adding these probabilities gives the number of events the model predicts for the cohort, on average. Comparing that expected number with the number of events estimated from the observed outcomes provides a compact summary of overall calibration: whether predicted risks, in aggregate, are too low, too high, or close to the observed event frequency.

The comparison is not as simple as counting events by the horizon. A participant censored before the horizon has an unknown event status at that horizon. Treating that participant as event-free can bias the observed count downward. The observed-to-expected ratio addressed here therefore uses inverse-probability-of-censoring weighting (IPCW) for the observed count, while the expected count is the sum of the model’s predicted risks. The target is the event by a prespecified horizon in a defined validation cohort, using predictions made at a common prediction time.

Definition: For validation participant $i$, let $T_i$ be the event time, $C_i$ the censoring time, $Y_i=\min(T_i,C_i)$, and $\Delta_i=I(T_i\le C_i)$, where $\Delta_i=1$ indicates an observed event. At horizon $\tau$, let $p_i(\tau)$ be the joint model’s predicted probability of the event by $\tau$, conditional on the information available at prediction time. The expected event count is $E(\tau)=\sum_{i=1}^n p_i(\tau)$. If $\widehat G(t\mid Z_i)$ estimates the probability of remaining uncensored through time $t$, conditional on censoring predictors $Z_i$, the IPCW observed event count is $$ \widehat O(\tau)=\sum_{i=1}^n \frac{\Delta_i I(Y_i\le \tau)}{\widehat G(Y_i^-\mid Z_i)}. $$ The censoring-adjusted observed-to-expected ratio is $\widehat O(\tau)/E(\tau)$, provided $E(\tau)>0$. Here $Y_i^-$ denotes the time just before $Y_i$, which accommodates the event-time censoring probability under the chosen convention for ties.

For a prediction made at a landmark time, the same construction applies to event risk over the specified future interval: define $p_i$ for that interval, and define the observed event and censoring times relative to the landmark among participants eligible for prediction then. Everyone in the comparison must have predictions for the same target interval and be included according to the same validation-cohort definition. Joint-model predictions may be conditional on the longitudinal measurements available at the landmark; future measurements or outcome information must not leak into the validation predictions.

Why the Censoring Weight Works

The weight for an observed event at time $t$ is the inverse of the probability that the participant would remain uncensored until just before $t$. An event observed despite a low probability of remaining uncensored represents more information about event occurrence in the target cohort than an event observed where censoring is rare. The weight compensates for this unequal observability; it does not change the model’s predicted risks.

The key assumption is conditional independent censoring: given the measured predictors $Z_i$, the event time and censoring time are independent. Also required is positivity: for covariate patterns and times relevant to events by $\tau$, the probability of remaining uncensored is positive and not so close to zero that weights become practically unstable. The censoring model must adequately estimate that probability. In the following justification, treat $G$ as the true censoring survival function; replacing it by an estimate adds estimation error.

Justification: Conditional on $T_i$ and $Z_i$, an event at time $T_i\le\tau$ is observed with probability $G(T_i^-\mid Z_i)$ under conditional independent censoring. Thus the conditional expected contribution to the weighted count is $$ E\!\left[\left.\frac{\Delta_i I(Y_i\le\tau)}{G(Y_i^-\mid Z_i)}\right|T_i,Z_i\right] =I(T_i\le\tau)\frac{G(T_i^-\mid Z_i)}{G(T_i^-\mid Z_i)} =I(T_i\le\tau). $$ If $T_i>\tau$, the contribution is zero. Therefore the weighted contribution has conditional expectation equal to the participant’s event indicator by $\tau$. Summing over participants shows why the IPCW count estimates the cohort’s number of events by the horizon, under the stated assumptions. This is an expectation argument, not a claim that the estimated count equals the unobserved true count in every sample.

The denominator $E(\tau)$ is the model’s expected count because each $p_i(\tau)$ is the predicted event probability for one participant. It is not multiplied by a censoring weight: the prediction targets event occurrence in the cohort, not whether the event happened to be observed. The same horizon and event definition must be used for both parts of the ratio.

Worked Example: Computing a Weighted Ratio

Suppose a hypothetical validation cohort has six participants and a common horizon of 8 months. The joint-model risks are $0.10, 0.20, 0.15, 0.25, 0.30,$ and $0.20$. Three participants have observed events by 8 months; their estimated probabilities of remaining uncensored to just before their event times are $0.80, 0.50,$ and $0.90$. Two other participants are censored before 8 months, and one is known to be event-free through 8 months. Assume the censoring model is appropriate and the required probabilities are positive.

Expected count. Sum the six predicted risks: $E(8)=0.10+0.20+0.15+0.25+0.30+0.20=1.20$ expected events.

Weighted observed count. The three observed events contribute $1/0.80=1.25$, $1/0.50=2.00$, and $1/0.90=1.111$. The participants censored before the horizon contribute no observed event term, and the participant known to be event-free through the horizon contributes zero. Therefore, $\widehat O(8)=1.25+2.00+1.111=4.361$.

Ratio and interpretation. The ratio is $4.361/1.20=3.634$. In this hypothetical cohort, the censoring-adjusted observed event count is about 3.63 times the model’s expected count. This indicates substantial aggregate underprediction at 8 months in this sample. It is not an estimate that each participant’s risk is underpredicted by that factor; nor does this small illustrative calculation establish that the discrepancy is statistically significant.

Interpreting the Ratio

A ratio of 1 means the estimated observed count equals the expected count. A ratio above 1 means more events are estimated to have occurred than the model predicted, indicating aggregate underprediction. A ratio below 1 means fewer events are estimated than predicted, indicating aggregate overprediction. For example, a ratio of 0.80 means the weighted observed count is 80% of the expected count; it does not mean every individual risk is 20% too high.

This is a calibration-in-the-large summary at one horizon, expressed as a count ratio. It does not assess whether predictions are well calibrated across the risk range, whether the model orders participants effectively, or whether a few prediction errors cancel many others. A ratio near 1 can coexist with important errors in subgroups or across risk levels. It is also specific to the validation cohort, outcome definition, prediction time, and horizon.

Worked Example: A Ratio Near One

In a second hypothetical validation cohort, the summed joint-model risks at a fixed horizon are $E(\tau)=6.30$. Four observed events have censoring survival probabilities $0.80, 0.75, 0.60,$ and $0.70$ at their event times. Assume conditional independent censoring given the covariates in the censoring model, adequate positivity, and a correctly specified censoring model.

The weighted observed count is $\widehat O(\tau)=1/0.80+1/0.75+1/0.60+1/0.70=1.250+1.333+1.667+1.429=5.679$. Thus the ratio is $5.679/6.30=0.9014$.

The weighted count is about 90% of the expected count, so this cohort shows modest aggregate overprediction at this horizon. The ratio is not exactly one, but the numerical difference alone does not establish whether it reflects more than sampling variability; that requires uncertainty assessment.

Assumptions, Diagnostics, and Alternatives

The calculation is only as credible as its censoring adjustment. If censoring depends on event risk after conditioning on $Z_i$, the observed events may not represent participants with the same covariate values who were censored. If the censoring model omits important predictors, the weights can fail to correct that selection. If estimated censoring probabilities are very small, a few observed events can receive enormous weights and dominate $\widehat O(\tau)$. A zero estimated probability makes the formula undefined.

Useful checks include reviewing the censoring model’s predictors and fit, inspecting the distribution of the weights among observed events, and examining whether the largest weights drive the result. Confirm that censoring probabilities are estimable through the horizon and that the event and censoring definitions are correct. A horizon with little follow-up support can make the estimate unstable even if the arithmetic is correct. Weight truncation can reduce the influence of extreme weights, but it changes the estimator and can introduce bias; any such choice should be reported and its impact examined.

A naïve observed count that includes only events actually seen by the horizon ignores events that may have occurred after early censoring. A Kaplan–Meier estimate of event-free survival can instead provide an estimated event probability under its own censoring assumptions; it is an alternative way to estimate an overall event frequency, rather than the event-by-event weighted count shown here. IPCW has the practical advantage of allowing censoring adjustment conditional on measured predictors, but depends on a censoring model and can be variable. Neither aggregate approach replaces examining calibration across predicted risks, and neither ratio alone evaluates discrimination.

Worked Example: Why the Naïve Count Can Mislead

Suppose a third hypothetical validation cohort has two observed events by the horizon, with estimated censoring survival probabilities of $0.50$ and $0.25$ at the event times. The model’s summed predicted risks are $E(\tau)=5.00$. Assume conditional independent censoring given the variables used for the censoring model and positive censoring probabilities at both event times.

The naïve count is 2, giving a naïve observed-to-expected ratio of $2/5.00=0.40$. The IPCW count is $1/0.50+1/0.25=2+4=6$, giving an IPCW ratio of $6/5.00=1.20$. The naïve calculation suggests marked overprediction, while the weighted calculation suggests modest underprediction. The difference arises because the observed events occurred where censoring made event observation relatively unlikely. This example illustrates why censoring adjustment matters; the large weight of 4 also signals that the weighted estimate may be unstable.

Common Pitfall and Practical Reporting

Wrong reasoning: “The ratio is above 1, so every joint-model risk is too low by the same proportion.” The ratio compares two cohort-level counts; it does not identify individual errors or show how calibration varies across risk levels. Correction: report it as an aggregate comparison at the stated horizon, then use appropriate complementary calibration assessments to investigate the pattern of predictions. Do not interpret the ratio as a causal effect or as a probability that the model is correct.

A clear report states the prediction time and horizon, the event definition, the validation population, the predicted-risk sum, the censoring-adjusted observed count, the estimated censoring model and its conditioning variables, and the resulting ratio. Also describe relevant diagnostics, especially extreme weights and weak follow-up support. An observed-to-expected ratio is undefined when the expected count is zero, and a very small expected count can make the ratio numerically volatile. The ratio is a useful summary, not a substitute for checking assumptions or quantifying uncertainty.

Check Your Understanding

Answer each question, then compare with the solutions below.

A hypothetical cohort has predicted risks summing to 4.0 and an IPCW observed count of 5.0. Calculate and interpret the ratio.
An observed event has estimated censoring survival probability $0.40$ at its event time. What is its IPCW contribution?
Why does a participant censored before the horizon not simply receive an event-free status in the observed count?
Give two assumptions or diagnostics relevant to an IPCW observed count.

Solutions. (1) The ratio is $5.0/4.0=1.25$; the weighted observed count is 125% of the expected count, indicating aggregate underprediction at that horizon. (2) The contribution is $1/0.40=2.5$. (3) The participant’s event status at the horizon is unknown; treating censoring as event-free can undercount events. (4) Examples include conditional independent censoring given measured predictors, positivity of censoring survival probabilities, adequate censoring-model specification, and checking whether a few extreme weights dominate the estimate.