Integrated AUC And Horizon Weighting

Why Integrate AUC Across Horizons?

A transition-specific AUC describes how well a prediction score ranks cases above controls at a particular horizon. But a model may rank well at short horizons and less well at longer ones. Reporting a single horizon can therefore hide variation across the period in which predictions will be used. An integrated AUC summarizes discrimination across a prespecified range of horizons; its interpretation depends on both that range and the weights assigned to different times.

This summary is not a universal property of a model. It is a weighted average of horizon-specific discrimination for a chosen prediction origin, transition, target population, horizon interval, and definition of cases and controls. Changing any of those choices can change the estimand, not just its numerical estimate.

The case-control definition must be held fixed across horizons and follow the chosen competing-transition convention. For example, competing transitions may be treated as controls or handled another prespecified way; those choices define different discrimination targets. The scores must also refer to the same target transition and prediction origin. AUC values based on different populations or incompatible case-control definitions should not be combined as though they described one curve.

Choosing the Horizon Weights

The weight function expresses which parts of the horizon range should contribute most to the summary. With \(w(h)=1\), every unit of elapsed time receives equal weight. This is a natural choice when the entire interval is of equal interest, but it does not give equal weight to a set of irregularly spaced evaluation points unless the integration accounts for their spacing.

Other choices can emphasize horizons that matter more for a clinical decision or represent a target distribution of use. A weight based on the frequency of decisions at different horizons answers a different question from uniform time weighting. Weights can also be specified to emphasize early or late prediction, but should be chosen before examining model performance where possible. They must be nonnegative for the summary to remain a weighted average of AUC values.

A weighted summary remains between the smallest and largest AUC values over the interval, provided the weights are nonnegative and the denominator is positive. It does not, however, reveal whether performance is steady or whether a high summary is driven by a narrow part of the range. Always report the horizon interval and weighting scheme alongside the integrated value.

Calculating an Integrated AUC From a Horizon Grid

In practice, AUC values are usually estimated at a finite set of horizons. The integral can then be approximated numerically. With uniform weights and a grid \(h_1<\cdots<h_m\), the trapezoidal rule uses the width of each interval: $$ \widehat{\operatorname{iAUC}}_{\mathrm{uniform}} = \frac{ \displaystyle\sum_{j=1}^{m-1} (h_{j+1}-h_j) \frac{\widehat{\operatorname{AUC}}(h_j)+\widehat{\operatorname{AUC}}(h_{j+1})}{2} } {h_m-h_1}. $$ If \(w(h)\) varies, approximate the numerator by integrating \(w(h)\widehat{\operatorname{AUC}}(h)\) and the denominator by integrating \(w(h)\), using the same numerical scheme. With a grid of prespecified interval weights \(q_j\), a discrete weighted average is \(\sum_j q_j\widehat{\operatorname{AUC}}(h_j)/\sum_j q_j\). These are related summaries, but the discrete weights and the integration rule should be stated explicitly.

A short justification clarifies what the calculation does. When \(w(h)\geq0\), define the normalized weight density \(g(h)=w(h)/\int_{h_L}^{h_U}w(u)\,du\). Then \(g(h)\geq0\), its integral is one, and $$ \operatorname{iAUC}_{w}=\int_{h_L}^{h_U}g(h)\operatorname{AUC}(h)\,dh. $$ Because this is an average with nonnegative weights, if \(a\leq\operatorname{AUC}(h)\leq b\) throughout the interval, then \(a\leq\operatorname{iAUC}_{w}\leq b\). Thus the integrated value summarizes the curve; it cannot improve the underlying discrimination or erase poor performance at a particular horizon.

How Weighting Changes the Answer

An integrated AUC is easiest to interpret when its weights correspond to the question being asked. A uniform time average and an average weighted toward near-term decisions are both legitimate summaries, but they are not interchangeable. Differences between them reveal where the model performs better or worse relative to the intended use.

For continuous horizon weighting, the same principle applies, but weights are assigned across time rather than only at selected points. In particular, a weight proportional to the frequency of clinical decisions at each horizon yields an average for that decision distribution, if that distribution is defined and normalized over the analysis interval. The weight function should not be confused with the estimator's censoring weights: the former defines which horizons matter in the summary, whereas censoring weights address incomplete outcome observation when estimating horizon-specific discrimination.

Estimation, Diagnostics, and Limitations

The integration step does not by itself solve the problem of estimating AUC under right censoring. At each horizon, use an estimator appropriate to the outcome-observation process and the chosen competing-transition definition. If inverse probability of censoring weighting (IPCW) is used, its assumptions and diagnostics apply to each horizon-specific estimate. In particular, censoring must be adequately addressed under the estimator's assumptions, and estimated censoring probabilities must not be so small that weights become unstable. Integrating unstable estimates can produce a precise-looking number that is still unreliable.

Before reporting an integrated value, check the following:

• Case and control support: At each included horizon, verify that the data provide cases and controls under the chosen definition. If either group is absent, the horizon-specific AUC is undefined; a numerical integration cannot repair that.
• Comparable targets: Confirm that the origin, target transition, target population, and competing-transition convention are consistent across horizons.
• Censoring and weights: Inspect the horizon-specific estimates and any censoring weights for instability. Sparse risk sets, very large weights, or abrupt changes in the estimates warrant investigation.
• Grid and interpolation: Include enough horizons to represent plausible changes in discrimination. A coarse grid may miss a sharp rise or fall between evaluation points.
• Weight sensitivity: Compare the prespecified summary with a small number of substantively reasonable alternatives. Large changes indicate that the single integrated value depends strongly on the weighting choice.

A common mistake is to take an unweighted arithmetic mean of AUC estimates at irregularly spaced horizons and call it a uniform time-integrated AUC. For example, the grid in the first worked example has intervals of 1 and 2 years, so giving all three points equal weight overrepresents the short interval relative to the long one. The correction is to integrate over time, accounting for interval widths, or to state explicitly that the target is a discrete average over equally weighted evaluation points rather than a uniform time average.

Another limitation is that AUC concerns ranking. A model can have a high AUC while systematically overpredicting transition probabilities; it can also have a lower AUC while providing useful calibrated probabilities. Integrated AUC does not measure calibration or overall prediction error. The integrated Brier score is a distinct summary based on prediction error across horizons, while a single-horizon AUC focuses on ranking at one time. The appropriate measure depends on whether the validation question concerns discrimination, calibration, or probability accuracy.