Reporting Transition-Specific Validation Results

Why reporting details are part of validation

A validation result such as “the model performed well” is not interpretable on its own. In a multi-state setting, a prediction may refer to a particular state at a specified future time, conditional on being in a particular state at the prediction origin. Change the origin, horizon, target state, or eligible population, and the validation question changes. A result that omits these details can make two unlike analyses look comparable.

A useful report therefore makes the prediction target reconstructible before presenting a score. It also states how incomplete follow-up was handled, what each metric measures, and how uncertainty was estimated. This tutorial focuses on that reporting discipline; it does not replace the earlier material on transition prediction, calibration, or discrimination.

Definition: A prediction origin is the time at which a prediction is made, along with the state and observed history that make a subject eligible for it. A horizon is the elapsed time from that origin to the time at which the prediction is evaluated. For an origin at time $s$, a transition-probability prediction for state $j$ at horizon $u$, conditional on being in state $g$ at $s$, targets $$ p_{gj}(s,u\mid \mathcal{H}(s)) = P\{X(s+u)=j\mid X(s)=g,\mathcal{H}(s)\}, $$ where $X(t)$ is the subject’s state at time $t$ and $\mathcal{H}(s)$ is the recorded history available at the prediction origin.

This definition concerns the probability of occupying state $j$ at the horizon. It is not automatically the probability of making a direct $g\to j$ transition by that time. For example, a prediction of being dead two years after an illness-free origin can include paths through illness. State explicitly which target is validated. If the target is a transition probability, name both the origin state and the destination state, and clarify whether the prediction is conditional on the origin state or applies to a broader population.

What a complete validation result should specify

A reader should be able to identify the validation sample, reconstruct who contributed to each result, and understand what the reported number estimates. Report the following items in the main text, a table, or a clearly referenced supplement:

Population and design: setting, calendar period, inclusion criteria, number of subjects, and whether validation is temporal, external, or otherwise separated from model development. State the number eligible for each origin and transition.
Prediction definition: the prediction origin, eligible state, horizon in meaningful units, target destination, and whether the outcome is state occupancy or a direct transition. If predictions are updated over time, state how origins were selected and whether one subject can contribute multiple predictions.
Follow-up and competing events: how follow-up ends, how much follow-up is available at the horizon, and which transitions compete with the target. A competing transition is part of the multi-state outcome, not ordinary loss to follow-up.
Censoring method: name the method, the variables used to model censoring if applicable, and the assumptions needed for its use. In particular, state the conditional independent-censoring assumption and positivity requirement for inverse-probability weighting, or describe the assumptions behind the chosen alternative.
Metrics: define each metric sufficiently to establish its target, scale, and direction of better performance. Give the number of subjects or prediction records contributing and the method used to account for censoring.
Uncertainty and display: state how intervals or standard errors were obtained, the resampling unit if bootstrap methods were used, and what the calibration plot’s points, curve, bands, and denominators represent.

For a transition-probability calibration result, report predicted probabilities and censoring-adjusted observed probabilities on the same scale. A calibration plot should identify the transition and horizon in its caption or axes; give the number at risk or otherwise explain the number contributing across the prediction range; and show uncertainty where feasible. If grouped predictions are used, state how groups were formed. If a smooth curve is used, describe the smoothing method and any adjustment for censoring. The plot should not imply that observed proportions are raw proportions when they have been adjusted.

Censoring adjustment and what the score means

Suppose the target is whether a subject occupies a specified state at a fixed horizon. If some subjects are censored before that horizon, their state at the horizon is unknown. Simply treating them as not having the target event changes the outcome definition; dropping them without justification can also distort the validation sample. A censoring-adjusted method is needed when censoring is present and the estimand concerns the full eligible validation population.

For example, an inverse-probability-of-censoring weighted Brier score compares predicted probabilities with binary horizon outcomes, giving more weight to outcomes that were less likely to be observed. In general terms, if $Y_i$ is the horizon outcome when observed, $R_i$ indicates that it is observable, and $w_i$ is the appropriate inverse observation-probability weight, the weighted mean squared error has the form

\widehat{BS} = \frac{\sum_i R_i w_i(Y_i-\hat p_i)^2} {\sum_i R_i w_i}.

The precise weights depend on the outcome and censoring scheme. For this approach to represent the target population, censoring must be independent of the relevant outcome conditional on the variables used in the censoring model, and the probability of remaining observable must be positive in the relevant covariate and follow-up regions. Those assumptions should be assessed and reported, not treated as guaranteed by use of a weighting formula.

Justification: In a simplified fixed-horizon setting, let $R$ indicate whether the outcome $Y$ is observed and let $G(Z)=P(R=1\mid Z)$, where $Z$ contains the variables that make censoring conditionally independent of $Y$. If $G(Z)>0$, then $$ E\left\{\frac{RY}{G(Z)}\mid Y,Z\right\} = \frac{Y\,E(R\mid Y,Z)}{G(Z)} = \frac{Y\,P(R=1\mid Z)}{G(Z)} = Y. $$ Thus inverse-probability weighting can recover the outcome contribution in expectation under the stated assumptions. For multi-state outcomes, the actual weighting and observable-outcome rules must match the target and event history; this simplified identity is not a substitute for specifying the estimator used.

Alternatives include complete-case summaries, pseudo-observation methods, and other censoring-adjusted estimators. They do not become interchangeable merely because they produce similar numbers: each has its own estimand, assumptions, and uncertainty procedure. Report the estimator by name and enough detail to identify its implementation. When using complete cases, explain why the resulting subset estimates the intended target; when using pseudo-observations, state how they were constructed and how uncertainty was handled.

Worked reporting examples

Worked Example: Reporting a Transition-Specific Result

Hypothetical validation. An illness-death model is evaluated in 500 external-validation subjects who are alive and illness-free at baseline. The target is the probability of occupying the illness state two years after baseline; death before two years is a competing transition. The mean predicted probability is 0.18. A censoring-adjusted estimate of the observed illness-state probability is 0.20, with a 95% confidence interval of 0.16 to 0.24. The reported two-year Brier score is 0.143 and the transition-specific AUC is 0.71.

The calibration difference, defined here as observed minus mean predicted probability, is $$ 0.20-0.18=0.02. $$ Thus the observed estimate is 2 percentage points higher than the mean prediction. The confidence interval for the observed probability includes 0.18, but that fact alone is not a formal test of the calibration difference: an interval for the difference would require its own uncertainty calculation.

A useful report would say: “Among 500 external-validation subjects alive and illness-free at baseline, the mean predicted probability of occupying the illness state at two years was 18%. The censoring-adjusted observed probability was 20% (95% CI 16% to 24%); death was treated as a competing transition. The two-year Brier score was 0.143, and the transition-specific AUC was 0.71. Confidence intervals were obtained by resampling subjects.” The report should additionally name the censoring-adjustment estimator and specify the AUC estimator and interval method. The Brier score is a mean squared probability error on a zero-to-one scale, with smaller values indicating better accuracy; the AUC describes discrimination for the specified target, not calibration.

Worked Example: Showing the Censoring Adjustment in a Brier Score

Hypothetical illustration. Four subjects have a known binary state-occupancy outcome at the specified horizon; a fifth subject was censored before the horizon and has no known outcome. For the four observable outcomes, the predictions, outcomes, and appropriate inverse-observation weights are: $(0.7,1,1.25)$, $(0.2,0,1.25)$, $(0.6,1,2)$, and $(0.3,0,2)$. The weights correspond to estimated observation probabilities of 0.8, 0.8, 0.5, and 0.5. Assume, for illustration, that the censoring model and positivity condition are adequate for these records.

The squared errors are $(1-0.7)^2=0.09$, $(0-0.2)^2=0.04$, $(1-0.6)^2=0.16$, and $(0-0.3)^2=0.09$. Therefore, $$ \widehat{BS} = \frac{1.25(0.09)+1.25(0.04)+2(0.16)+2(0.09)} {1.25+1.25+2+2} = \frac{0.6625}{6.5} = 0.1019. $$

This illustrative weighted score is 0.1019, or about 0.102 on the zero-to-one Brier scale. The censored subject is not assigned outcome zero: the target outcome is unknown. In an actual analysis, the weights must be estimated using a justified censoring model, and the report should say how that model was specified and checked. This small calculation illustrates the arithmetic, not evidence that the weighting assumptions hold.

Worked Example: Reporting Uncertainty for a Calibration Difference

Hypothetical result. In an external validation, the estimated two-year observed-minus-predicted probability difference for a specified transition is 0.030. A subject-level bootstrap with 1,000 resamples gives an estimated standard error of 0.015. If a normal-approximation interval is used, it is $$ 0.030\pm 1.96(0.015) = 0.030\pm 0.0294 = (0.0006,\ 0.0594). $$

The result can be reported as a 3.0-percentage-point observed excess, with a 95% normal-approximation interval from 0.06 to 5.94 percentage points. This interval is close to zero at its lower endpoint, so the data support a small positive difference under this interval procedure, but the estimate remains imprecise. The interval describes uncertainty in the calibration difference under the analysis and resampling assumptions; it is not a probability statement about the fixed difference.

The report should state that subjects, rather than individual prediction records, were resampled. If each subject contributes several prediction origins, resampling records independently would incorrectly treat within-subject observations as independent. Also state whether the interval is normal-approximation, percentile, or another bootstrap interval; the method matters, especially with skewed estimates or boundary values.

Diagnostics, comparisons, and limitations

A concise table can make results easier to compare without hiding the target:

Reporting item	What readers need to know
Prediction target	Origin state, destination or target state, horizon, and occupancy versus direct-transition definition
Validation sample	Eligible subjects and prediction records for that target; whether records repeat within subjects
Censoring	Estimator, assumptions, variables in the censoring model, and amount of incomplete follow-up
Performance	Metric definition, scale, direction of better performance, estimate, and uncertainty
Calibration display	Observed and predicted quantities, adjustment method, group or smoother construction, and uncertainty

Before finalizing results, inspect whether censoring weights are extreme, whether estimated observation probabilities approach zero, and whether the effective information is concentrated in a few subjects. Report weight summaries or a sensitivity analysis when these features are material. For plots, check that sparse regions are visible rather than smoothed into apparent certainty; report the number contributing to groups and avoid implying precise calibration where data are thin. Also verify that event counts and denominators agree across the text, tables, and figures.

A Brier score summarizes overall probability accuracy and is influenced by outcome frequency; it does not isolate calibration from discrimination. AUC summarizes ranking ability for its specified target but does not say whether probabilities are numerically accurate. Calibration plots and summaries address agreement between predicted and observed risks, but their appearance depends on the adjustment and smoothing choices. These measures answer different questions, so report them together when appropriate rather than treating one as a replacement for the others.

Transition-specific results can be unstable when few subjects enter an origin state, when few target transitions occur, or when follow-up is limited at the chosen horizon. A single average score can conceal poor performance in subgroups or at particular horizons. Report relevant subgroup or time-specific results when they were prespecified and sufficiently supported, and avoid presenting exploratory contrasts as definitive. A validation report describes predictive performance in the evaluated population; it does not establish causal effects or guarantee performance in a different population.

Common pitfall: Reporting “the two-year risk of illness was 20%” without saying who was at risk at the prediction origin, what state was predicted, or how deaths and censoring were handled. The correction is to name the origin state and eligible cohort, define the two-year target as a state-occupancy probability or direct-transition risk, identify competing transitions, and state the censoring-adjusted estimator. Otherwise, the number has no uniquely interpretable validation target.

A practical reporting sequence

Define the prediction task.
Name the origin, eligible state and history, horizon, target state or transition, and eligible population.

Describe what was observed.
Give sample and event counts, follow-up availability, competing transitions, and the number contributing to each analysis.

Specify estimation and uncertainty.
Identify censoring adjustment, metric definitions, assumptions, interval method, and resampling unit.

Interpret with limits.
State the result in context and units, distinguish calibration from discrimination, and flag sparse data or assumption concerns.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A report gives an AUC of 0.74 for “two-year progression.” Name two details that must be added to define the prediction target.
Why is it generally wrong to code a subject censored before the horizon as a non-event for a state-occupancy outcome?
A calibration difference is reported as $0.06$, defined as observed minus predicted. Interpret its sign and units if probabilities are on the zero-to-one scale.
What should be resampled in a bootstrap when each person contributes predictions from several origins, and why?

Solutions. (1) State the prediction origin and eligible state, the exact target state or transition, and whether “progression” means occupancy at two years or a direct transition by two years; also specify the eligible population. (2) The horizon state is unknown after early censoring, so coding it as a non-event systematically changes the target outcome. Use a justified censoring-adjustment method or explain why another estimator is appropriate. (3) The observed probability is estimated to be 6 percentage points higher than the mean predicted probability; this is a calibration discrepancy, not a causal effect. (4) Resample subjects, retaining all their prediction records together, so within-subject dependence is represented rather than treating repeated records as independent observations.