Two Different Questions About Transition Predictions
A transition prediction can rank people well while giving probabilities that are too high or too low. Conversely, a model can predict the average frequency of a destination accurately yet offer little help in distinguishing which individuals will reach it. These are not contradictory findings: calibration concerns the accuracy of absolute probabilities, whereas discrimination concerns how well predictions rank individuals with different outcomes.
Interpretation must begin with a precisely defined prediction target. Suppose a person is in state \(i\) at landmark time \(s\), and the prediction concerns their state at a fixed horizon \(t\), where \(t>s\). Let \(\hat p_{ij}(s,t\mid X)\) denote the model-predicted probability of being in state \(j\) at time \(t\), given covariates \(X\) measured for prediction. The matching observed binary outcome is \(Y_j=1\) if the person is in state \(j\) at \(t\), and \(Y_j=0\) otherwise. Other destinations, including competing absorbing states, are part of the \(Y_j=0\) group for this particular comparison.
This definition targets state occupancy at \(t\). If the scientific question is instead whether a particular transition occurred at least once before \(t\), that is a different outcome and requires predictions and validation measures for that event definition. Mixing the two targets makes calibration and discrimination difficult to interpret.
Assumptions Before Combining the Results
The validation data should represent the population to which the assessment is intended to apply, and predictions should be evaluated for the same landmark, horizon, origin state, and destination used to define the outcome. The model must be evaluated without allowing validation outcomes to influence its fitted predictions. If predictions have been adjusted or evaluated using resampling, the interpretation should also state whether the reported performance is apparent, internally corrected, or externally validated.
With complete outcome ascertainment at \(t\), the definitions can be assessed directly from observed states. When some individuals are censored before \(t\), simply classifying them as non-events is generally wrong: their state at \(t\) is unknown. Calibration and discrimination estimates must use methods that account for censoring and competing destinations, such as appropriate censoring-weighted or transition-probability estimation procedures. These procedures require assumptions appropriate to the estimator; for inverse-probability weighting, for example, censoring must be conditionally independent of the target outcome given the variables used in the censoring model, and the probability of remaining observed must be positive in the relevant groups. Diagnostics should check whether these conditions are plausible. A high-performing score cannot repair an invalid outcome estimate.
For a destination-specific assessment, competing destinations count as non-cases for the binary outcome \(Y_j\), not as ordinary independent censoring. For a multi-destination prediction, the destination probabilities for a given person must also be coherent: probabilities of mutually exclusive states at the same horizon should be nonnegative and sum to one over the exhaustive set of possible states. Calibration of one destination at a time does not by itself guarantee this joint coherence.
Calibration and Discrimination Measures
The population calibration condition for a predicted probability \(q\) is \(E(Y_j\mid q)=q\). In practice, calibration is examined by comparing predicted probabilities with estimated outcome frequencies across ranges of predictions, as well as by comparing overall mean prediction with overall observed risk. The latter is a useful calibration-in-the-large check, but agreement of the two overall averages does not establish agreement throughout the prediction range. Calibration curves, calibration intercepts and slopes, and subgroup checks provide additional views; none should be interpreted without considering uncertainty and the number of observed transitions.
For a fixed-horizon binary outcome, the area under the receiver operating characteristic curve (AUC) has a ranking interpretation. It is the probability that a randomly selected case receives a higher prediction than a randomly selected non-case, with half credit for a tie:
Here, \(Q_j\) denotes the predicted probability for destination \(j\). An AUC of 0.5 indicates no better pairwise ranking than random ordering under this definition; an AUC of 1 indicates that every case ranks above every non-case. AUC does not say whether probabilities such as 0.2 or 0.8 are numerically accurate. For censored data, the case and control comparisons must be estimated with a method suitable for the chosen time-dependent outcome and censoring pattern.
Worked Example: Perfect Ranking With Poor Absolute Probabilities
Consider a hypothetical validation sample of ten people, all in the same origin state at the landmark and fully observed at the horizon. Five are in destination \(j\) at the horizon and five are not. The model assigns 0.90 to each of the five cases and 0.60 to each of the five non-cases.
Discrimination. There are \(5\times5=25\) case-control pairs. In every pair, the case prediction, 0.90, exceeds the control prediction, 0.60. Thus
Calibration. The mean predicted probability is \((5(0.90)+5(0.60))/10=0.75\), whereas the observed proportion in destination \(j\) is \(5/10=0.50\). The overall mean prediction exceeds the observed frequency by 0.25, or 25 percentage points. Within the two prediction groups, the observed frequencies are 1.00 among those assigned 0.90 and 0.00 among those assigned 0.60. The model separates the outcomes perfectly in this small sample, but its absolute probabilities are inaccurate in both groups. The result supports strong ranking in these data, not accurate risk communication.
Why Ranking Can Stay the Same When Calibration Changes
A useful justification follows directly from the AUC definition. If every predicted probability \(q\) is transformed by the same strictly increasing function \(g\), then \(q_a>q_b\) if and only if \(g(q_a)>g(q_b)\). Ties are preserved as well. Therefore every case-control comparison has the same result after transformation, so the AUC is unchanged. But the transformed values need not agree with observed frequencies. Recalibration can therefore improve or worsen calibration without changing discrimination, provided it preserves the ordering.
Worked Example: A Monotone Change Alters Probabilities, Not AUC
For four hypothetical people, suppose the outcomes are \(0,0,1,1\) and the predicted probabilities are \(0.20,0.40,0.60,0.80\), respectively. Each of the two cases has a higher prediction than each of the two non-cases, so the four case-control pairs are all concordant and the AUC is \(4/4=1\). The mean prediction is \((0.20+0.40+0.60+0.80)/4=0.50\), equal to the observed outcome proportion \(2/4=0.50\).
Now apply the strictly increasing transformation \(g(q)=0.10+0.75q\). The predictions become \(0.25,0.40,0.55,0.70\). Their order is unchanged, so the AUC remains \(1\). Their mean is \((0.25+0.40+0.55+0.70)/4=0.475\), which is 0.025 below the observed proportion of 0.50. Even this small illustration shows that a ranking statistic cannot certify calibration. In a real validation sample, calibration must be judged across prediction ranges and with appropriate uncertainty, not from the mean alone.
Reading the Pair of Findings Together
Calibration and discrimination should be reported as complementary evidence, not blended into a single label such as “good prediction.” A useful interpretation records the destination and horizon, gives calibration evidence and its uncertainty, and reports discrimination using the matching case-control definition. It then states what each result supports and what it does not. The following patterns are especially informative:
| Calibration | Discrimination | Interpretation |
|---|---|---|
| Good | Good | Probabilities are reasonably accurate and cases tend to rank above non-cases. Check uncertainty and whether performance applies to the intended population. |
| Poor | Good | Ordering may be useful, but absolute probabilities should not be presented as accurate without suitable recalibration and fresh validation. |
| Good overall, weak by range | Weak | The average predicted risk may match the average observed risk while individual ranking adds little; inspect the calibration curve and prediction spread. |
| Poor | Poor | Neither absolute risk nor ranking is supported adequately by these results; investigate the target, data quality, model, and validation design. |
For multiple destinations, repeat the interpretation transition by transition. A high AUC for one destination does not imply high AUC for another, and separate good calibration results do not ensure that a vector of predicted probabilities is coherent. A macro-average across transitions can conceal a particularly poor destination-specific result, so the component results should remain visible.
Worked Example: A Calibrated Average Does Not Guarantee Useful Ranking
Suppose ten hypothetical validation subjects have complete horizon outcomes. Every subject receives a predicted probability of 0.20 for destination \(j\); two subjects are in \(j\) at the horizon and eight are not.
The mean predicted probability is \(10(0.20)/10=0.20\), and the observed proportion is \(2/10=0.20\), so the overall calibration-in-the-large comparison agrees exactly. However, all predicted values tie. There are \(2\times8=16\) case-control pairs, each receiving half credit, giving
The predictions provide no ranking within this sample. The matching overall average is useful evidence about the mean risk, but it cannot establish accurate conditional risks or individual-level predictive separation. With only ten subjects, uncertainty is also substantial; the calculation is an illustration, not evidence of population calibration.
Diagnostics, Alternatives, and Limits
A joint reading should prompt checks rather than end with two numbers. Look for sparse transitions or prediction ranges, influential subjects, and substantial uncertainty in estimated observed risks. Check whether calibration differs across clinically important subgroups and whether competing-destination probabilities remain coherent. For censored outcomes, examine the censoring model and the effective information available at the horizon. Large changes under reasonable choices of censoring model or validation procedure indicate that the apparent performance may be fragile.
The Brier score and log score are alternatives that evaluate probabilistic predictions using the predicted probabilities and observed outcomes together. They can be useful complementary summaries, but they do not replace separate calibration and discrimination assessments when the scientific question is whether probabilities are accurate and whether ranking is useful. AUC also has limitations: it is insensitive to strictly increasing transformations, does not measure the size of probability errors, and may be uninformative when predictions have little variation. None of these summaries alone establishes clinical utility, causality, or transportability to a population with a different case mix.
- Wrong reasoning: “The AUC is high, so the model's predicted transition probabilities are accurate.”
- Correction: AUC evaluates pairwise ordering, not numerical agreement. Check calibration for the same transition and horizon separately.
- Wrong reasoning: “The average prediction equals the observed event proportion, so the model is calibrated.”
- Correction: Overall agreement can hide systematic overprediction in one range and underprediction in another. Inspect calibration across the prediction distribution and report uncertainty.
- Wrong reasoning: “People censored before the horizon did not have the destination.”
- Correction: Their horizon state is unknown. Use an appropriate method for censoring and competing destinations rather than coding them as non-cases.
The central practical lesson is to keep the questions distinct: calibration asks whether a predicted probability corresponds to the observed chance of its specified transition outcome, while discrimination asks whether those who experience that outcome tend to receive higher predictions. A responsible conclusion states both findings, in context, and does not let strength in one dimension stand in for evidence about the other.
Check Your Understanding
Answer each question using a fixed destination and horizon unless the question says otherwise.
- A transition model has AUC 0.91, but predicted risks average 0.70 while the observed destination frequency is 0.40. What can and cannot be concluded?
- There are 12 cases and 8 non-cases. All case predictions exceed all non-case predictions. What is the empirical AUC?
- Why does applying the same strictly increasing transformation to every prediction leave the AUC unchanged?
- At the horizon, destination \(j\) and a competing absorbing destination are observed. For a destination-\(j\) binary outcome, how is the competing destination classified?
Solutions. (1) The AUC supports strong case-control ranking in the evaluated sample, but the 0.30 difference between mean prediction and observed frequency signals overprediction overall; calibration across prediction ranges and uncertainty still need assessment. (2) All \(12\times8=96\) pairs are concordant, so the empirical AUC is \(96/96=1\). (3) A strictly increasing transformation preserves every pairwise ordering and tie, so each term in the AUC definition is unchanged. (4) It is a non-case for the destination-\(j\) outcome at that horizon, while remaining a competing destination that must be handled correctly in estimating outcomes when censoring is present.