Summarizing Discrimination Across Prediction Horizons

Why Summarize Discrimination Across Horizons?

A prediction model for survival outcomes can rank patients differently at different prediction horizons. A marker may distinguish people who experience an event soon from those who remain event-free, yet be less useful for separating patients at a later horizon. Reporting only one time-dependent area under the ROC curve (AUC) can hide this pattern. Reporting a collection of AUC values preserves the pattern but can make it difficult to state one overall measure for a prespecified period.

A time-averaged AUC provides a compact summary over a chosen range, but it is not a substitute for examining the horizon-specific values. Its meaning depends on the time range, the ROC definition, and the weights assigned to different times. Those choices should be made before viewing validation results when the goal is confirmatory evaluation.

Definition: For a specified time-dependent ROC definition, let $A(t)$ denote the population AUC at prediction horizon $t$. A weighted average AUC over $[a,b]$ is $$ \overline{A}_{q,[a,b]}= \frac{\int_a^b q(t)A(t)\,dt}{\int_a^b q(t)\,dt}, \qquad 0<\int_a^b q(t)\,dt<\infty. $$ Here $q(t)$ is a prespecified, nonnegative time-weighting function. The numerator must be integrable; in particular, if $A(t)$ is measurable and bounded between 0 and 1, the stated finite integral of $q(t)$ ensures this. The summary is an average of discrimination over the chosen time range, weighted according to $q(t)$.

Specify What Is Being Averaged

The definition of $A(t)$ must be fixed before averaging. For example, under the cumulative-dynamic definition, cases at horizon $t$ are individuals with event time $T\le t$, and controls are individuals with $T>t$. If $R(t)$ is the predicted risk by time $t$, then the population AUC is

A_{\mathrm{CD}}(t) = P\{R_i(t)>R_j(t)\mid T_i\le t,\ T_j>t\} +\frac{1}{2}P\{R_i(t)=R_j(t)\mid T_i\le t,\ T_j>t\}.

The half-credit for ties makes the AUC equal to the probability that a randomly selected case receives a higher risk score than a randomly selected control, with tied scores counted as one-half. An incident-dynamic AUC uses a different case definition. These two AUC curves target different comparisons; their values should not be mixed within one summary as if they were estimates of the same function.

At each horizon, the risk score must correspond to that horizon. A model may assign a different risk to the same individual at different times, so $A(t)$ summarizes both the changing case-control definition and the model's horizon-specific ranking. Use the same target population, ROC definition, prediction setup, and stated time range when comparing models. If these differ, a contrast in summary AUC can reflect a change in target as well as a change in model discrimination.

In a validation sample with censoring, the horizon-specific AUC estimates must account for which event statuses are known. A censoring-adjusted estimator relies on assumptions appropriate to its censoring model, including adequate probability of remaining uncensored over the horizons being evaluated (positivity). If the required conditional independence assumption or censoring model is not credible, the estimated curve—and therefore its summary—may be biased. Averaging does not repair a biased horizon-specific estimator.

Choosing the Time Weights

If $q(t)=1$ over $[a,b]$, the result is the uniform time average: each unit of time has equal weight. This answers, “What is the average AUC across this interval when time itself is weighted uniformly?” It does not give equal weight to each patient, each event, or each observed prediction time.

A different nonnegative $q(t)$ can give more emphasis to times that matter more for a particular decision. For example, a prespecified function may assign greater weight to the first year than to later years. The denominator normalizes the weights, so multiplying all values of $q(t)$ by the same positive constant leaves the summary unchanged. The weights must still satisfy the integrability condition in the definition: a function that is zero throughout the interval, or has an infinite integral, does not define a normalized average.

A uniform average over calendar time is not the same as an event-weighted average. Weighting horizons according to the event-time distribution changes which times contribute most and gives a different estimand. Such a choice may be reasonable for a specified purpose, but it should be stated and prespecified rather than introduced after inspecting which weighting produces the most favorable value.

Estimating the Summary From a Grid of Horizons

In practice, horizon-specific AUC estimates are often available at a finite set of times. A continuous-time summary then requires numerical integration or a discrete, prespecified summary over those times. A common numerical approximation for the uniform time average is the trapezoidal rule. For grid points $a=t_1<\cdots<t_K=b$, it approximates the area under the AUC curve by summing trapezoid areas between adjacent estimates.

\widehat{\overline{A}}_{\mathrm{uniform}} \approx \frac{1}{b-a} \sum_{k=1}^{K-1} (t_{k+1}-t_k)\frac{\widehat{A}(t_k)+\widehat{A}(t_{k+1})}{2}.

This calculation assumes a linear change in the estimated AUC between adjacent grid points as an approximation. It is not a claim that the true curve is exactly linear. A sparse grid can miss sharp changes, while a dense grid may add computation without resolving the uncertainty in the curve. Report the grid and integration rule, and inspect the horizon-specific estimates alongside the summary.

Worked Example: Uniform Average by the Trapezoidal Rule

Suppose a hypothetical validation analysis estimates cumulative-dynamic AUC values of 0.78, 0.74, 0.68, and 0.62 at horizons 1, 2, 3, and 4 years. Estimate the uniform average over years 1 through 4 using linear interpolation between adjacent horizons.

The interval length is $4-1=3$ years. Each adjacent pair of horizons is one year apart, so the estimated area is

\begin{aligned} \widehat{\text{area}} &=1\left(\frac{0.78+0.74}{2}\right) +1\left(\frac{0.74+0.68}{2}\right) +1\left(\frac{0.68+0.62}{2}\right)\\ &=0.760+0.710+0.650\\ &=2.120\ \text{AUC-years}. \end{aligned}

Dividing by the three-year interval gives $2.120/3=0.7067$. The estimated uniform time-average AUC is therefore 0.7067 over years 1–4. The horizon-specific estimates also show a decline from 0.78 to 0.62; the average alone does not communicate that change.

Worked Example: Giving Greater Weight to Earlier Horizons

Use the same hypothetical AUC estimates and linear interpolation. Suppose the analysis plan assigns weight $q(t)=2$ from year 1 to year 2 and $q(t)=1$ from year 2 to year 4. This makes the first year count twice as much per unit time as the following two years.

The weighted denominator is

\int_1^4 q(t)\,dt=2(2-1)+1(4-2)=4\ \text{weighted years}.

Over years 1–2, the interpolated mean AUC is $(0.78+0.74)/2=0.76$, so the weighted contribution to the numerator is $2(1)(0.76)=1.52$. Over years 2–4, the mean AUC under linear interpolation is $(0.74+0.62)/2=0.68$, so that contribution is $1(2)(0.68)=1.36$. Thus

\widehat{\overline{A}}_{q,[1,4]} \approx \frac{1.52+1.36}{4}=0.720.

This weighted summary is higher than the uniform average of 0.7067 because the higher AUC values occur earlier and receive more weight. The difference does not show that one calculation is intrinsically better: the two summaries answer different questions. The weighting rule needs a rationale that is independent of the observed AUC pattern.

Discrete Prespecified Summaries

Sometimes a study has a finite set of clinically specified prediction horizons rather than a goal of averaging continuously over time. A discrete weighted summary can be defined using nonnegative weights $v_k$ with $\sum_{k=1}^K v_k>0$:

\widehat{A}_{\mathrm{disc}} = \frac{\sum_{k=1}^K v_k\widehat{A}(t_k)} {\sum_{k=1}^K v_k}.

This is an average over the selected horizons, not necessarily an approximation to the continuous-time average. In particular, assigning equal weights to a set of unequally spaced horizons gives equal importance to each selected horizon, not equal importance to each unit of time.

Worked Example: A Discrete Summary at Three Horizons

Consider hypothetical estimated AUC values of 0.80, 0.70, and 0.60 at prespecified horizons of 6 months, 2 years, and 5 years. Suppose the analysis assigns weights 2, 3, and 5, respectively, to represent the relative importance specified before validation.

The total weight is $2+3+5=10$. The weighted numerator is

$$ 2(0.80)+3(0.70)+5(0.60)=1.60+2.10+3.00=6.70. $$

Therefore, the discrete summary is $6.70/10=0.670$. This is a weighted average over the three selected horizons. It is not a uniform time average from 6 months to 5 years, because the weights were assigned to horizons rather than to the lengths of the intervals between them.

How to Read, Compare, and Diagnose a Summary

The summary AUC is on the same 0-to-1 scale as the horizon-specific AUCs. A value of 0.5 corresponds to chance-level ranking for the specified case-control comparison, while a value of 1 represents perfect ranking. A summary such as 0.71 means that the weighted average of the horizon-specific ranking probabilities is 0.71 under the stated ROC definition and weighting scheme. It does not mean that the model has AUC 0.71 at every horizon.

For a fair comparison of two prediction models, use the same horizons, target population, ROC definition, censoring-adjustment approach, and time weights. The difference between their summaries then compares discrimination averaged under that shared specification. Also show the horizon-specific AUCs, preferably as a table or curve, because models can have similar averages but different time patterns. For example, one model may be better early and worse late, while another has steadier discrimination.

Several checks help identify a summary that may be misleading:

Check the horizon-specific curve. Look for sharp changes, reversals in model rankings, or estimates near the ends of the range that are based on limited information.
Check support under censoring. At late horizons, few individuals may remain uncensored or few controls may remain event-free. Censoring adjustment cannot create information where support is inadequate.
Check grid sensitivity. If estimates change substantially when the integration grid is refined or the interval endpoints are altered slightly, the reported average may be too dependent on approximation or range choice.
Check that the estimand is consistent. Do not combine cumulative-dynamic and incident-dynamic AUCs in one curve or average without explicitly defining a new target.
Check the weighting rationale. Confirm that the time range and weights were chosen for the intended decision, not selected because they yield a preferred result.

A point estimate does not show its sampling uncertainty. If inference about a summary is needed, uncertainty should be assessed using an approach that reflects how the horizon-specific estimates are jointly estimated. Treating estimates at different horizons as independent can be inappropriate because they are computed from overlapping individuals and outcomes. The summary alone also says nothing about calibration: a model may rank patients well while its predicted risks are systematically too high or too low.

Common Mistakes and Alternatives

Choice or mistake	What it means	How to report or correct it
Reporting only one AUC at a selected horizon	Answers a single-horizon question but may conceal later changes	Show the horizon-specific curve and add a prespecified summary if useful
Uniform time average	Weights equal lengths of time equally	State the interval and numerical integration rule
Discrete weighted average	Weights selected horizons, not automatically all intervening time	State the horizons and weights explicitly
Choosing weights after seeing results	Can make the summary favor observed performance	Prespecify the target interval and weighting rationale
Assuming the average describes every horizon	Hides possible increases, decreases, or reversals	Display and interpret horizon-specific AUCs alongside the average

A single prespecified horizon is often preferable when one decision time is the sole target. A curve is preferable when the timing of discrimination changes is central. A weighted summary is useful when an overall measure across a specified period is needed, provided that the curve remains available for interpretation. These approaches are complementary rather than interchangeable.

Key takeaway: A summary across prediction horizons is defined not just by an averaging formula but by the ROC estimand, time interval, and weighting rule. State each one, verify that the weights have a finite positive integral, and retain the horizon-specific curve so the average does not obscure changes over follow-up.

Check Your Understanding

Answer each question using the stated estimand and weighting scheme. Brief solutions follow the questions.

Why does a nonnegative weighting function alone fail to define a normalized average?
For a uniform average on years 0–2, estimated AUC is 0.70 at year 0 and 0.60 at year 2. Using linear interpolation, calculate the average.
At three equally selected horizons, the AUC values are 0.90, 0.70, and 0.50. What is their equally weighted discrete average, and why is it not automatically the continuous-time average?
Two models have the same average AUC over years 1–5, but one has higher early AUC and lower late AUC. What should be reported in addition to the average?
Why should cumulative-dynamic and incident-dynamic AUC estimates not be combined in an average without further definition?

Solutions. (1) The denominator might be zero or infinite. Require $0<\int_a^b q(t)\,dt<\infty$, along with an integrable numerator. (2) Under linear interpolation, the area is $2(0.70+0.60)/2=1.30$ AUC-years; dividing by 2 years gives $0.650$. (3) The average is $(0.90+0.70+0.50)/3=0.700$. It weights the selected horizons equally, regardless of spacing, so it need not weight time uniformly. (4) Report the horizon-specific AUC curve or values, using the same horizons and estimand for both models, so the reversal in relative performance is visible. (5) Their case definitions differ, so the AUCs refer to different ranking comparisons and do not form one consistent function $A(t)$ unless a new combined estimand is explicitly defined.