Time-Dependent AUC For A Transition

What a Transition-Specific AUC Measures

A transition prediction assigns each eligible subject a predicted probability of making a specified transition within a chosen time horizon. For example, at a common prediction origin, a model might give each subject a probability of moving from state $i$ to state $j$ within the next five years. The time-dependent AUC asks a focused question: does the prediction tend to rank subjects who experience the target transition above subjects who do not meet the chosen comparison definition?

An AUC is not a measure of whether predicted probabilities are numerically accurate. A model can rank subjects well while systematically overpredicting, or predict probabilities well on average while ranking them only moderately. Calibration and Brier scores address different aspects of prediction performance; the AUC isolates ranking ability at the specified horizon.

Definition: Fix a prediction origin, a target transition $i\to j$, and a horizon $\tau$. Let $S_r(\tau)$ be subject $r$'s predicted probability of that transition by $\tau$, and let $Y_r(\tau)$ be the binary outcome defined for the validation target. A subject with $Y_r(\tau)=1$ is a case; a subject with $Y_r(\tau)=0$ is a control. The horizon-specific AUC is $$ \operatorname{AUC}_{ij}(\tau) = P\{S_{\mathrm{case}}(\tau)>S_{\mathrm{control}}(\tau)\} +\frac{1}{2}P\{S_{\mathrm{case}}(\tau)=S_{\mathrm{control}}(\tau)\}. $$

The case and control labels must come from a clearly stated validation target, not from the AUC formula itself. The prediction origin, the transition, the horizon, and the rule defining the binary outcome must be held fixed across the subjects being compared. In a multi-state setting, a subject may experience other transitions before $\tau$; the target definition must specify how those observed histories contribute to $Y_r(\tau)$. The calculations below use already classifiable binary outcomes. They do not resolve alternative conventions for competing transitions.

The score $S_r(\tau)$ is usually the model's predicted transition probability at the horizon. AUC uses only the ordering of scores: replacing every score by the same strictly increasing transformation leaves the ranking, and therefore the AUC, unchanged. Consequently, the AUC does not assess whether a score of 0.70 corresponds to a 70% event probability.

Empirical AUC From Complete Outcomes

When every subject's outcome at $\tau$ is known, the empirical AUC is a pairwise average. Compare each case's score with each control's score. Award one point when the case score is greater, zero when it is lower, and half a point for a tie.

\widehat{\operatorname{AUC}}_{ij}(\tau) = \frac{1}{n_1 n_0} \sum_{r:Y_r(\tau)=1} \sum_{s:Y_s(\tau)=0} \left[ I\{S_r(\tau)>S_s(\tau)\} +\frac{1}{2}I\{S_r(\tau)=S_s(\tau)\} \right],

Here $n_1$ and $n_0$ are the numbers of cases and controls, respectively, and $I\{\cdot\}$ equals 1 when the condition is true and 0 otherwise. The denominator $n_1n_0$ is the number of case-control pairs. This estimate is defined only if the validation data include at least one case and at least one control.

Worked Example: Pairwise Ranking at Five Years

Suppose a hypothetical validation sample has two cases for a specified transition by five years and three controls under a prespecified outcome definition. Their predicted five-year transition probabilities are:

Cases: 0.82 and 0.65. Controls: 0.71, 0.40, and 0.25.

There are $2\times3=6$ pairs. The case with score 0.82 outranks all three controls, contributing 3 wins. The case with score 0.65 loses to 0.71 but outranks 0.40 and 0.25, contributing 2 wins. There are no ties. Therefore,

\widehat{\operatorname{AUC}}_{ij}(5) = \frac{3+2}{6} = \frac{5}{6} = 0.8333.

In this hypothetical sample, a randomly selected case-control pair is ranked correctly about 83% of the time. This describes discrimination for this transition and five-year horizon under the stated outcome definition; it is not an estimate of the probability that any individual will make the transition.

Worked Example: Giving Ties Half Credit

Consider another hypothetical validation sample with two cases scored 0.70 and 0.50, and two controls scored 0.70 and 0.40. There are four pairs. The 0.70 case ties the 0.70 control, earning 0.5, and beats the 0.40 control, earning 1. The 0.50 case loses to 0.70, earning 0, and beats 0.40, earning 1. Thus,

\widehat{\operatorname{AUC}}_{ij}(\tau) = \frac{0.5+1+0+1}{4} = 0.625.

Counting the tie as a full win would incorrectly raise the estimate to 0.75. Half credit corresponds to treating equal scores as an unresolved ordering rather than evidence that the case ranks above the control.

Right Censoring and Weighted Pair Comparisons

If follow-up ends before the horizon, the subject's outcome at $\tau$ may be unknown. Simply excluding every censored subject can distort the AUC: inclusion may then depend on follow-up duration, which can be associated with predictors or outcome risk. Inverse probability of censoring weighting (IPCW) can adjust pair contributions when its assumptions are plausible. The earlier material on Diagnosing Censoring Weights In Transition Validation is relevant for checking the estimated censoring probabilities and weight behavior.

For a basic IPCW construction, suppose each case's target outcome is observed when the target transition occurs by $\tau$, and a control's status is known when observation continues through $\tau$. Let $C_r$ be the censoring time, and let $G(t\mid Z_r)=P(C_r\ge t\mid Z_r)$ be the conditional probability of remaining uncensored through time $t$, given measured covariates $Z_r$. A case whose transition is observed at time $T_r\le\tau$ receives weight $w_r=1/G(T_r-\mid Z_r)$; a control observed through $\tau$ receives weight $w_r=1/G(\tau\mid Z_r)$. These expressions assume that the status is ascertainable in the way just described; an endpoint that makes a status known earlier requires weights appropriate to its ascertainment time.

The weighted AUC is the weighted proportion of case-control pairs in which the case outranks the control, with half credit for ties:

\widehat{\operatorname{AUC}}_{\mathrm{IPCW}}(\tau) = \frac{ \displaystyle\sum_{r\in\mathcal C} \sum_{s\in\mathcal N} w_r w_s \left[I\{S_r(\tau)>S_s(\tau)\} +\frac{1}{2}I\{S_r(\tau)=S_s(\tau)\}\right] }{ \displaystyle\sum_{r\in\mathcal C} \sum_{s\in\mathcal N} w_r w_s },

Here $\mathcal C$ and $\mathcal N$ are the observed cases and controls with classifiable status under the specified endpoint. The weights compensate for the chance that a subject with a comparable observed history would have been censored before their status could be determined. The product $w_rw_s$ weights each pair; the denominator is the total pair weight.

Assumptions for the IPCW interpretation: The censoring model must adequately describe the probability of remaining uncensored, conditional on the covariates used in $G$. Conditional on those covariates, censoring must be independent of the outcome process in the sense needed to recover the classifiable case and control distributions. There must be a positive probability of remaining observed to each relevant ascertainment time, and the weights must be estimable with adequate precision. The prediction must be available at the stated origin, and the validation outcomes and scores must refer to the same transition and horizon.

Worked Example: IPCW Pairwise Calculation

In hypothetical validation data, suppose two observed cases have scores 0.80 and 0.60 and weights 2 and 1. Their censoring survival probabilities at their transition times are 0.50 and 1.00, respectively. Two controls have scores 0.70 and 0.30 and weights 1 and 2, corresponding to censoring survival probabilities of 1.00 and 0.50 through the five-year horizon. Assume the endpoint statuses are classifiable as described above and the censoring model is valid.

The total weight of case-control pairs is the product of the total case weight and total control weight: $ (2+1)(1+2)=9 $. For the case scored 0.80, the weighted pair contributions against controls 0.70 and 0.30 are $2\times1=2$ and $2\times2=4$, both wins. For the case scored 0.60, comparison with 0.70 is a loss, contributing 0, while comparison with 0.30 is a win, contributing $1\times2=2$. Hence,

\widehat{\operatorname{AUC}}_{\mathrm{IPCW}}(5) = \frac{2+4+0+2}{9} = \frac{8}{9} = 0.8889.

The weighted estimate is about 0.889. It need not equal the unweighted AUC from the observed subjects, because pairs involving subjects who were less likely to remain observed receive more weight. The estimate is credible only to the extent that the censoring model and its assumptions are credible.

Why the Pairwise Formula Works—and Where It Can Fail

The population definition is the probability of a correct ordering for a randomly selected case-control pair, with half credit for a tie. In a complete validation sample, each observed pair supplies one value: 1, 0, or 0.5. Averaging those values gives the empirical formula. With censoring, IPCW replaces each pair's equal contribution with a weight that reflects how likely its members were to have classifiable outcomes. Dividing by the sum of pair weights normalizes the weighted average. Under the stated assumptions and a consistently estimated censoring distribution, this targets the same pairwise ranking quantity for the specified endpoint.

Several checks help reveal when the calculation is not answering the intended question:

Confirm the target before comparing scores. Mixing prediction origins, horizons, or transition definitions changes the question. A score for a one-year transition probability is not directly the five-year score required here.
Check both groups and the score range. Very few cases or controls, or nearly identical scores, can make the estimate unstable or uninformative. If one group is absent, the AUC is undefined.
Inspect censoring weights and support. Small estimated $G(t\mid Z)$ values produce large weights. They can make a few pairs dominate the result; a lack of follow-up support cannot be repaired by weighting alone.
Check status ascertainment and censoring-model fit. Incorrectly classifying a censored subject as a control, or using a censoring model that omits important predictors of censoring, can bias the estimate. Weight diagnostics are useful but cannot establish conditional independence by themselves.
Look for information leakage. Scores must be generated using information available at the prediction origin. Using later observations to construct a predictor gives an invalidly optimistic ranking assessment.

The AUC ranges from 0 to 1. A value of 0.5 indicates no better pairwise ranking than an uninformative score in the relevant comparison, while 1 indicates that every case-control pair is ordered correctly. Values below 0.5 indicate that the ordering is worse than its reversal, though this may reflect a reversed score, a poorly performing model, or a mismatch between the score and the chosen endpoint. An AUC near 1 does not establish accurate absolute risks, good calibration, or clinical usefulness at a particular decision threshold.

Compared with calibration, the AUC ignores the numerical agreement between predicted and observed transition probabilities. Compared with the Brier score, it does not combine discrimination and probability accuracy into a subject-level squared-error measure. Unlike a threshold-based classification measure, it summarizes rankings across all possible score cutoffs. Each approach answers a different question; none makes the others unnecessary. AUC is also horizon-specific: it should not be read as a summary over time.

Common pitfall: “The AUC is 0.83, so the model predicts each subject’s transition probability accurately.” This is incorrect: the AUC concerns the ordering of case-control pairs, not the closeness of probabilities to outcomes. Assess probability accuracy with calibration and suitable prediction-error measures, and report the transition, horizon, and outcome definition alongside the AUC.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A fixed-horizon validation sample has one case scored 0.90 and two controls scored 0.90 and 0.30. Calculate the empirical AUC.
Why is an AUC not defined when the validation sample contains no controls?
A subject is censored before the horizon, so their target status cannot be determined. Why can treating that subject as a control be problematic?
A model has a high AUC but systematically predicts probabilities that are too large. Which aspect of performance does the AUC describe, and which does it miss?

Solutions. (1) The case ties the first control, earning 0.5, and outranks the second, earning 1. There are two pairs, so the AUC is $(0.5+1)/2=0.75$. (2) An AUC averages comparisons between cases and controls. With no controls, there are no such pairs and the denominator is zero. (3) Censoring means the outcome at the horizon is unknown, not that the subject is a control. Misclassification can alter the observed ranking comparison, especially if censoring relates to predictors or outcomes. (4) The AUC describes case-control ranking at the stated horizon; it does not assess whether predicted probabilities agree with observed transition frequencies. Calibration is needed to assess that agreement.

Time-Dependent AUC For A Transition

What You'll Learn