Two Different Questions About Discrimination
A risk score may rank people well for whether they have experienced an event by a given time, yet rank them differently for who is about to experience an event among those still event-free. These are distinct prediction questions. The cumulative-dynamic ROC curve evaluates the first: cases have experienced the event by time \(t\), while controls remain event-free after \(t\). The incident-dynamic ROC curve focuses on events occurring at, or within a specified short interval after, \(t\), comparing those cases with people who remain event-free through that interval.
That distinction matters in joint modeling. A prediction may use a patient’s longitudinal history up to a prediction time, and the score may be updated as new measurements arrive. If the practical question is who among the current risk set is most likely to fail soon, an incident-dynamic assessment targets that question more directly than a cumulative assessment. Neither approach is universally preferable: the right choice depends on the prediction target.
The interval formulation makes event status operational when time is recorded in days, months, or years rather than at a mathematical instant. As \(\Delta\) becomes very small, the incident case group approximates people failing around \(t\), and dynamic controls approximate people who survive past \(t\). For continuous event times, the instantaneous formulation is expressed conditionally on failure near \(t\), because the probability of an event at exactly one time point is zero.
When the score comes from a longitudinal model, \(M_i(t)\) should be computed from the history available by \(t\), for example the biomarker measurements through \(t\), and should target risk after \(t\). It must not use measurements collected after the prediction time or information revealed only after the event. A score could be a conditional event probability over \((t,t+\Delta]\), a predicted hazard near \(t\), or another risk ranking. The chosen score and its direction must be specified.
What the ROC Curve and AUC Mean
For a threshold \(c\), classify a person as high risk when \(M_i(t)\ge c\). Incident sensitivity is the probability that an interval case exceeds the threshold; dynamic specificity is the probability that a dynamic control falls below it. Varying \(c\) traces the incident-dynamic ROC curve, with sensitivity on the vertical axis and one minus specificity on the horizontal axis. The threshold convention for ties should be kept consistent.
The AUC summarizes ranking across all possible thresholds. Its pairwise probability interpretation follows directly from the ROC definition. Imagine drawing one incident case and one dynamic control independently. For a fixed case score \(m\), the chance the control score is below \(m\) is the control score distribution evaluated below \(m\). Averaging that chance over the case score distribution gives the probability that the case outranks the control. If scores can tie, assigning half credit to ties gives the conventional rank-based AUC.
Thus an AUC of 0.5 represents chance-level ranking under this case-control comparison, while an AUC of 1 indicates that every case score exceeds every control score, apart from any tie convention. An AUC below 0.5 can indicate that the score direction is reversed, though it can also reflect genuinely poor ranking. The AUC does not tell whether predicted probabilities are numerically accurate: discrimination and calibration are different properties.
Worked Interval Comparisons
Worked Example: Ranking Cases and Controls in a One-Year Interval
Consider six hypothetical candidate records evaluated at \(t=2\) years for the interval \((2,3]\). Their event times \(T\) and risk scores \(M(2)\) are shown below. Larger scores indicate greater risk. Assume event status is observed through year 3 for every record.
| Individual | Event time \(T\) (years) | Score \(M(2)\) | Status for \((2,3]\) |
|---|---|---|---|
| A | 2.4 | 0.80 | Incident case |
| B | 2.8 | 0.55 | Incident case |
| C | 3.4 | 0.70 | Dynamic control |
| D | 4.0 | 0.40 | Dynamic control |
| E | 1.0 | 0.95 | Not at risk at \(t=2\) |
| F | 5.0 | 0.30 | Dynamic control |
E is excluded because \(T=1.0\) means the individual had already experienced the event before the evaluation time. A and B are cases because their events fall in \((2,3]\); C, D, and F are controls because their events occur after year 3. There are \(2\times3=6\) case-control pairs.
A, with score 0.80, outranks all three controls, contributing 3 wins. B, with score 0.55, outranks D and F but not C, contributing 2 wins. There are no ties, so
In this hypothetical sample, a randomly selected incident case has a higher score than a randomly selected dynamic control in about 83% of pairs. This is a ranking summary for the specified risk set and one-year interval, not a claim that any individual’s event probability is 83%.
Worked Example: Why Cumulative-Dynamic and Incident-Dynamic AUCs Differ
Suppose all six scores below were calculated at the same baseline time and then held fixed. For the incident-dynamic comparison, evaluate at \(t=2\) with \(\Delta=1\): two individuals fail in \((2,3]\), two had failed by year 2 and are excluded from the risk set, and two remain event-free after year 3. For the cumulative-dynamic comparison, evaluate at year 3.
| Group at year 3 | Scores |
|---|---|
| Incident cases: events in \((2,3]\) | 0.90, 0.60 |
| Earlier cases: events by year 2 | 0.20, 0.50 |
| Controls: event-free beyond year 3 | 0.70, 0.40 |
For incident-dynamic AUC, compare only the two incident cases with the two controls. The score 0.90 wins against both controls; 0.60 wins against 0.40 but not 0.70. Therefore,
For cumulative-dynamic AUC at year 3, all four people who failed by year 3 are cases, including the two earlier cases. There are eight case-control pairs. Scores 0.90 and 0.60 contribute two and one wins; 0.20 contributes none; and 0.50 beats 0.40 but not 0.70, contributing one. Hence
The different results are not a contradiction. The incident-dynamic calculation asks whether the score separates people failing during the current interval from those surviving through it. The cumulative-dynamic calculation asks whether the score separates everyone who has failed by year 3 from those who survive beyond year 3. Earlier cases with low scores affect the cumulative comparison but not the incident comparison. Holding the score time fixed here ensures that the groups are being compared using the same information.
Worked Example: Giving Half Credit for Tied Scores
Suppose two interval cases have scores 0.70 and 0.50, while two dynamic controls have scores 0.70 and 0.30. There are four pairs. The first case ties one control and beats the other, contributing \(0.5+1=1.5\) pair credits. The second case loses to 0.70 and beats 0.30, contributing 1 credit. Thus
Counting the tied pair as either a full win or a full loss would make the result depend arbitrarily on how a tie is ordered. Half credit is the standard pairwise convention.
Assumptions, Data Checks, and Failure Modes
The definition is a population comparison, but its interpretation depends on how the prediction and validation data were constructed. Before interpreting an estimated curve or area, check the following conditions.
- Correct risk set: everyone considered for either group must be event-free at the start \(t\). A person whose event occurred before \(t\) cannot be an incident case or a dynamic control at \(t\).
- Correct interval labels: events in \((t,t+\Delta]\) are cases; individuals event-free beyond \(t+\Delta\) are controls. With these definitions, a person failing in the interval is not a control.
- Prediction-time discipline: scores must use information available by \(t\). Using post-\(t\) biomarker values or event information creates leakage and can inflate apparent discrimination.
- Comparable score meaning: scores should be oriented so that larger values mean greater event risk. When comparing different definitions or times, ensure the scores are based on a clearly specified prediction time and target.
- Known status: a person censored before the end of the interval may have unknown case or control status. Such a record cannot simply be labelled a control because no event was observed before censoring.
If censoring occurs before the interval endpoint, estimating the population AUC requires methods that account for incomplete event status and their associated assumptions. In particular, censoring adjustment relies on an adequately specified censoring model and sufficient probability of remaining observed in relevant groups. A complete-case calculation can be misleading if the people with fully known outcomes differ systematically from those censored. Competing events also require an explicit target: treating a competing event as ordinary event-free follow-up changes the estimand and may not answer the intended clinical question.
Interval width is part of the target, not merely a computing detail. A wider \(\Delta\) includes more events and evaluates prediction over a longer period; it may yield a different ordering and a different AUC. A very narrow interval may better approximate instantaneous discrimination but contain few cases, making estimates unstable. The chosen width should reflect the decision or clinical question, and should be reported.
How It Differs From the Main Alternative
| Feature | Incident-dynamic | Cumulative-dynamic |
|---|---|---|
| Cases | Events in the interval after \(t\) | Events by time \(t\) |
| Controls | Event-free through the interval | Event-free beyond \(t\) |
| Question | Who among those currently at risk fails soon? | Who has experienced an event by \(t\), versus who has not? |
| Emphasis | Near-term, risk-set discrimination | Discrimination for accumulated event status |
Cumulative-dynamic analysis is often natural when the decision concerns whether a person will have experienced an event by a fixed horizon. Incident-dynamic analysis is more directly aligned with near-term triage among those still event-free. A standard binary ROC analysis can be appropriate when outcomes are fully observed at one prespecified endpoint, but it does not by itself resolve censoring or changing risk sets in time-to-event data.
Incident-dynamic AUC also has limitations. It is conditional on surviving event-free to \(t\), so the composition of the risk set changes over time. Its value at one time need not represent performance at another time, nor does a high AUC establish accurate absolute risk predictions. In joint-model validation, the longitudinal history and the prediction landmark must therefore be stated alongside the event interval. These considerations help prevent a discrimination measure from being interpreted as a general measure of model quality.
Common Mistakes
- Including people who failed before \(t\) in the risk set. They cannot experience an incident event after \(t\); exclude them from both interval cases and controls.
- Calling someone a control because no event was observed. If follow-up ended before \(t+\Delta\), the person’s status through the interval may be unknown.
- Interpreting the incident AUC as cumulative performance. The incident measure concerns events in a specified interval among the current risk set, not all events accumulated since baseline.
- Changing the prediction information without noticing. A score updated at \(t\) and a score fixed at baseline are different predictors. State which is being evaluated and do not use information recorded after \(t\).
- Treating AUC as calibration. AUC assesses ranking, not whether predicted risks match observed event probabilities.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- At \(t=4\), an individual had an event at \(T=3.5\). Can this person be an incident case or dynamic control for \((4,5]\)?
- For \((4,5]\), classify a person with \(T=4.7\) and a person with \(T=5.6\), assuming event status is observed through year 5.
- Two cases have scores 0.8 and 0.4; two controls have scores 0.6 and 0.3. With no ties, calculate the pairwise AUC.
- Why can incident-dynamic and cumulative-dynamic AUCs differ at the same time point?
- Why is a person censored at \(4.6\) not automatically a dynamic control for \((4,5]\)?
Solutions. (1) No. The event occurred before \(t=4\), so the person is not in the risk set. (2) \(T=4.7\) is an incident case because it lies in \((4,5]\); \(T=5.6\) is a dynamic control because it is beyond 5. (3) The 0.8 case beats both controls, and the 0.4 case beats 0.3 but not 0.6. The AUC is \(3/4=0.75\). (4) Cumulative cases include events before the interval, while incident cases include only events within it; those groups can have different score distributions. (5) Censoring at 4.6 leaves event status through year 5 unknown, so the person cannot be classified as event-free through the interval merely because no event was observed before censoring.