Discrimination of Landmark Predictions by Horizon

What Does Discrimination Mean at a Landmark?

A landmark prediction is made for people who have survived, remained event-free, or otherwise met the eligibility criteria up to a specified landmark time. At that time, a model uses the information available to predict future event risk. A discrimination measure asks whether people predicted to be at higher risk tend, in fact, to experience the event sooner than people predicted to be at lower risk. For clinical use, the question is often tied to a particular prediction horizon: for example, whether a model ranks patients who experience the event within the next year above those who remain event-free for at least a year.

That horizon-specific question differs from asking whether a model generally orders people by their residual time to event. The first compares event-by-horizon cases with event-free-through-horizon controls. Its answer can change with the horizon, even when the prediction model and landmark cohort are unchanged.

Definition: Let \(s\) be the landmark time. For subject \(i\) who is eligible at \(s\), let \(T_i^L\) be the time from \(s\) to the event, \(C_i^L\) the time from \(s\) to censoring, \(Y_i=\min(T_i^L,C_i^L)\), and \(\delta_i=I(T_i^L\le C_i^L)\). For a fixed horizon \(h>0\), a case has \(T_i^L\le h\), and a control has \(T_i^L>h\). If \(Z_i\) denotes information available at the landmark, a model may produce predicted risk \(\hat r_i(h)=1-\hat S_i^L(h\mid Z_i)\), where \(\hat S_i^L(h\mid Z_i)\) is predicted survival through \(h\) after the landmark.

The target population is the landmark-eligible cohort, not everyone originally enrolled. Predictions must use information available at the landmark, and the outcome window begins there. This is consistent with the conditioning set used in landmark prediction; the earlier material on calibration of landmark risk at fixed horizons concerns whether predicted risks agree with observed event frequencies. Discrimination instead assesses ranking.

The Horizon-Specific Area Under the ROC Curve

For a randomly selected case and a randomly selected control from the landmark population, the horizon-specific area under the receiver operating characteristic curve, or AUC at horizon \(h\), is the probability that the case receives a higher predicted risk than the control, with half-credit for a tie:

\operatorname{AUC}(h) = P\{\hat r_{\mathrm{case}}(h)>\hat r_{\mathrm{control}}(h)\} +\frac{1}{2}P\{\hat r_{\mathrm{case}}(h)=\hat r_{\mathrm{control}}(h)\}.

An AUC of 0.5 represents no better-than-chance ranking in this case-control comparison; an AUC of 1 means every case has a higher predicted risk than every control. The AUC does not say whether predicted risks are numerically accurate. It also does not imply that a high-risk prediction causes an event.

With complete event status through \(h\), let \(\mathcal{D}_h\) be the cases and \(\mathcal{Q}_h\) the controls. If there are \(n_1\) cases and \(n_0\) controls, the empirical AUC is the average of the pairwise comparison scores:

\widehat{\operatorname{AUC}}(h) = \frac{1}{n_1n_0} \sum_{i\in\mathcal{D}_h} \sum_{j\in\mathcal{Q}_h} \left[ I\{\hat r_i(h)>\hat r_j(h)\} +\frac12 I\{\hat r_i(h)=\hat r_j(h)\} \right].

This calculation requires both case and control status to be known. A subject censored before \(h\) has unknown status at \(h\): the subject may later experience the event by the horizon or may remain event-free beyond it. Simply discarding such subjects can bias discrimination estimates, particularly when censoring is related to predictors or outcome risk.

Worked Example: Computing an AUC With a Tied Score

Suppose a hypothetical validation set contains two cases and two controls at horizon \(h\). The cases have predicted risks 0.80 and 0.60; the controls have predicted risks 0.70 and 0.60. Assume all four subjects have known status through \(h\).

There are \(2\times2=4\) case-control pairs. The case with risk 0.80 outranks both controls, contributing \(1+1=2\). The case with risk 0.60 loses to the control with risk 0.70 and ties the control with risk 0.60, contributing \(0+\frac12=0.5\). Thus

\widehat{\operatorname{AUC}}(h) = \frac{2+0.5}{4} = 0.625.

In this hypothetical sample, a randomly chosen case-control pair receives a correctly ordered score, counting ties as half a correct ordering, in 62.5% of pair comparisons. This is a sample discrimination estimate, not a statement about the probability that any individual patient will experience the event.

Why Discrimination Depends on the Horizon

Changing \(h\) changes who counts as a case and who counts as a control. It can also change the prediction itself, since \(\hat r_i(h)\) is a risk over a particular time interval. Consequently, the AUC at one horizon cannot be assumed to describe ranking at another. Evaluations should report the horizon and the landmark time, and should use the risk predictions that correspond to that horizon.

Worked Example: Ranking at Two Different Horizons

Consider four hypothetical subjects in a landmark cohort with event times, measured from the landmark, of 0.5, 1.5, 4, and 6 time units. Assume complete follow-up through both horizons. At \(h=1\), only the first subject is a case. Their predicted risks at \(h=1\), in the same subject order, are 0.80, 0.40, 0.60, and 0.20. The case outranks two of the three controls, so

\widehat{\operatorname{AUC}}(1)=\frac{2}{3}=0.667.

At \(h=3\), the first two subjects are cases and the remaining two are controls. Suppose the corresponding predicted risks at \(h=3\) are 0.70, 0.90, 0.50, and 0.30. Each case outranks both controls, giving four correctly ordered pairs out of four:

\widehat{\operatorname{AUC}}(3)=\frac{4}{4}=1.000.

The perfect estimate at \(h=3\) comes from this small hypothetical set and does not establish perfect ranking in a population. The example shows why a model’s discrimination needs a stated horizon: the case-control comparison itself changes when the horizon changes.

Handling Censoring With Inverse-Probability Weights

A common approach to censoring is inverse-probability-of-censoring weighting (IPCW). Define the conditional censoring survival function \(G(t\mid Z_i)=P(C_i^L\ge t\mid Z_i)\). Assume that censoring is independent of the event time conditional on the variables \(Z_i\) used to model censoring, and that there is a positive probability of remaining uncensored long enough to determine status at the horizon for every relevant covariate pattern. In symbols, the key requirements are \(T_i^L\perp C_i^L\mid Z_i\) and \(G(h+\mid Z_i)>0\) in the target population. For event times, use the left-limit convention \(G(T_i^L-\mid Z_i)\); with continuous censoring times, this is the probability of being uncensored through the event time.

An observed case has \(\delta_i=1\) and \(Y_i\le h\), so \(Y_i=T_i^L\). Its probability of being observed as an event by \(h\), conditional on its event time and \(Z_i\), is \(G(T_i^L-\mid Z_i)\). It receives case weight \(w_i=1/G(T_i^L-\mid Z_i)\). An observed control has \(Y_i>h\), which establishes that the subject was event-free and uncensored through \(h\). Its control weight is \(v_i=1/G(h+\mid Z_i)\). Subjects censored before \(h\) do not have a known case or control status and do not enter these observed groups.

Under the stated assumptions, weighting corrects for the unequal chance of observing the required status. A weighted estimate compares the weighted case and control populations:

\widehat{\operatorname{AUC}}_{\mathrm{IPCW}}(h) = \frac{ \displaystyle\sum_{i:\delta_i=1,\,Y_i\le h} \sum_{j:Y_j>h} w_i v_j \left[I\{\hat r_i(h)>\hat r_j(h)\} +\frac12 I\{\hat r_i(h)=\hat r_j(h)\}\right] }{ \displaystyle \left(\sum_{i:\delta_i=1,\,Y_i\le h}w_i\right) \left(\sum_{j:Y_j>h}v_j\right) }.

The denominator is the total weighted mass of all observed case-control pairs. The estimator is meaningful only if the censoring model is adequate and weights are not dominated by a few observations. If censoring is independent of event time without conditioning on covariates, a marginal censoring survival estimate may suffice; otherwise the censoring model should account for variables that explain censoring.

Worked Example: A Small IPCW Calculation

In a hypothetical validation set, two observed cases have predicted risks 0.90 and 0.60 and event times 0.5 and 1.5. Two observed controls, each known event-free through \(h=2\), have predicted risks 0.70 and 0.30. Assume continuous censoring times and a correctly specified conditional censoring model. It gives \(G(0.5-\mid Z)=0.80\), \(G(1.5-\mid Z)=0.50\), and \(G(2\mid Z)=0.40\) for both controls.

The case weights are \(1/0.80=1.25\) and \(1/0.50=2.00\). Each control weight is \(1/0.40=2.50\). The first case outranks both controls, contributing \(1.25(2.50+2.50)=6.25\) to the weighted numerator. The second case outranks only the control with risk 0.30, contributing \(2.00(2.50)=5.00\). The weighted denominator is

$$ (1.25+2.00)(2.50+2.50)=3.25(5.00)=16.25. $$

Therefore,

\widehat{\operatorname{AUC}}_{\mathrm{IPCW}}(2) = \frac{6.25+5.00}{16.25} = 0.692.

This value is the censoring-weighted ranking estimate under the assumed censoring model; it is not the unweighted fraction of pairs in the observed sample. The calculation would be unreliable if the censoring probabilities were misspecified or close to zero.

Justification, Diagnostics, and Alternatives

The weighting logic follows from an observation-probability argument. For a case with event time \(T_i^L\le h\), conditional independent censoring implies that the probability the event is observed is \(G(T_i^L-\mid Z_i)\). If \(A_i\) indicates that the event is observed, then, conditional on the event time and \(Z_i\),

E\left[\frac{A_i}{G(T_i^L-\mid Z_i)}\mid T_i^L,Z_i\right] = \frac{P(A_i=1\mid T_i^L,Z_i)}{G(T_i^L-\mid Z_i)} = 1.

Likewise, for a true control with \(T_i^L>h\), the probability its control status is observed is \(G(h+\mid Z_i)\), so the expected observed-control indicator divided by this probability is also one. Thus, in expectation, the weights restore the case and control contributions that censoring would otherwise hide. Applying the corresponding case and control weights to each pair estimates the target pairwise ranking probability. This justification depends on the conditional independence and positivity assumptions; weights cannot repair unmeasured informative censoring or a model that assigns zero observation probability to part of the target population.

Useful diagnostics include the numbers of observed cases and controls, the fraction censored before \(h\), the estimated censoring survival probabilities, and the distribution of IPCW weights. Very large weights indicate weak positivity and can make the estimate unstable. Check that there are enough cases and controls to form informative pairs, that predicted risks vary, and that the evaluation predictions were produced without using validation outcomes. For uncertainty, resampling should preserve the subject as the sampling unit; when predictions are estimated from data, the validation design must also account for how the model was trained.

A residual-time concordance measure such as Harrell’s concordance index addresses a different question: it summarizes ordering over comparable pairs with differing event times, rather than comparing cases and controls defined by one fixed horizon. The two measures can disagree without either calculation being erroneous. The Brier score and calibration measures assess prediction error or agreement between predicted and observed risk, not just ordering. A model can rank well but systematically overpredict risk, or be well calibrated while separating cases and controls only modestly.

Horizon-specific AUC is therefore useful when a clinical decision is tied to a defined time window, but it has limits. It does not quantify clinical benefit, identify a useful treatment threshold, or assess the accuracy of absolute risk estimates. AUC estimates may be imprecise when few events occur by the horizon, and censoring adjustment becomes unstable when observation probabilities are small. With competing risks, the case definition and the treatment of competing events must be specified for the intended prediction target; the simple event-free control definition and formulas here do not automatically settle that choice.

Common Mistakes

Leaving the horizon unstated. An AUC without a specified landmark and horizon does not identify which cases and controls were compared. Report both times and the relevant risk set.
Treating early censoring as a control. A person censored before \(h\) is not known to be event-free through \(h\). Use an appropriate censoring adjustment or another valid analysis rather than assigning that person control status.
Calling AUC a measure of calibration. AUC concerns ordering, not whether a predicted risk such as 0.30 corresponds to a 30% event frequency. Assess calibration separately.
Confusing horizon discrimination with residual-time concordance. One compares case-control status at a selected \(h\); the other summarizes orderings across event times. State which target the analysis estimates.
Ignoring extreme IPCW weights. Large weights can make an estimate highly variable. Examine the censoring model, positivity, and weight distribution before interpreting the point estimate.

Key takeaway: Horizon-specific discrimination asks whether a landmark model assigns higher predicted risk to people who experience the event by \(h\) than to people who remain event-free beyond \(h\). With censoring, that ranking must be estimated under explicit assumptions and with suitable adjustment for the unequal chance of observing case or control status.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

At a fixed horizon, a case has predicted risk 0.55 and a control has predicted risk 0.55. How does this pair contribute to the empirical AUC?
Why should a subject censored before the horizon not automatically be counted as a control?
At horizon \(h\), there are three cases and two controls, with complete outcome information. Across the six pairs, four cases outrank controls, one pair is tied, and one is discordant. What is the empirical AUC?
What is the purpose of dividing an observed case’s contribution by \(G(T_i^L-\mid Z_i)\)?
Give one reason a horizon-specific AUC may differ from a residual-time concordance measure.

Solutions. (1) The tied pair contributes \(1/2\) to the numerator. (2) Censoring before \(h\) leaves event status at \(h\) unknown; the subject could experience the event before or after the horizon. (3) The tie contributes one-half, so the AUC is \((4+0.5)/6=0.750\). (4) The inverse probability weights for the chance that the event was observed despite censoring, under the conditional independent-censoring assumption. (5) The AUC compares cases and controls defined at one horizon, whereas residual-time concordance compares event-time ordering over comparable pairs at varying times.