Concordance For Evaluating Omics Risk Scores

What Concordance Measures

An omics survival model may produce a risk score for each patient. Concordance asks a focused question: when two patients can be ordered by their observed survival information, does the model assign a higher risk score to the patient who experiences the event earlier? It evaluates ranking, not the accuracy of predicted survival probabilities at a particular time.

Let \(T_i\) be the event time and \(C_i\) the censoring time for patient \(i\). The observed time is \(Y_i=\min(T_i,C_i)\), and the event indicator is \(\delta_i=I(T_i\le C_i)\). A patient with \(\delta_i=0\) is right-censored: the event time is known only to exceed \(Y_i\). Let \(r_i\) be the model’s risk score, with larger values indicating greater predicted risk and, ordinarily, shorter predicted survival. If a model instead outputs a survival score for which larger values mean longer survival, its direction must be reversed before applying the definitions below.

Definition: Under the strict pair convention used here, an ordered pair \((i,j)\) is comparable if \(\delta_i=1\) and \(Y_i<Y_j\). Then patient \(i\) is known to have had the event at \(Y_i\), while patient \(j\) is known to have remained event-free beyond \(Y_i\). The pair is concordant if \(r_i>r_j\), discordant if \(r_i<r_j\), and tied in risk score if \(r_i=r_j\).

This convention excludes pairs with tied observed times. Other software may apply a specified rule to tied event times, so a reported statistic should be accompanied by its tie convention. For a tie in risk score, the usual concordance contribution is one half: the model has not ordered the two patients either correctly or incorrectly.

Harrell’s Concordance

Harrell’s concordance, often called Harrell’s C-index, is the fraction of comparable observed pairs that are concordant, with half-credit for risk-score ties. It is calculated directly from the observed times and event indicators:

\widehat C_{\mathrm{Harrell}} = \frac{ \displaystyle\sum_{i\ne j} I(\delta_i=1,\ Y_i<Y_j)\,c(r_i,r_j) }{ \displaystyle\sum_{i\ne j} I(\delta_i=1,\ Y_i<Y_j) }, \qquad c(r_i,r_j)= \begin{cases} 1,&r_i>r_j,\\ \frac12,&r_i=r_j,\\ 0,&r_i<r_j. \end{cases}

The denominator is the number of comparable ordered pairs. The numerator adds one for each correctly ranked pair, one half for each tied score, and zero for each incorrectly ranked pair. The statistic lies between 0 and 1 when there is at least one comparable pair. A value near 0.5 indicates little ranking ability in the evaluated pairs; a value below 0.5 can indicate that the score direction has been reversed. Neither value by itself establishes how accurate a model’s numerical risk predictions are.

Worked Example: Calculating Harrell’s C

Suppose five hypothetical patients have the following observed survival data. Risk scores are on an arbitrary scale, with larger values intended to predict earlier events.

Patient	Observed time \(Y_i\) (months)	Event indicator \(\delta_i\)	Risk score \(r_i\)
A	4	1	0.90
B	6	0	0.80
C	8	1	0.30
D	5	1	0.70
E	10	0	0.20

Patient A had an observed event at month 4. Patients B, C, D, and E all have \(Y_j>4\), so these are four comparable pairs. A’s score of 0.90 exceeds each of their scores, giving four concordant pairs.

Patient D had an event at month 5. The later observed times are those of B, C, and E, giving three more comparable pairs. D’s score is lower than B’s (discordant), but higher than C’s and E’s (two concordant pairs). Patient C had an event at month 8 and is comparable with E, whose observed time is 10 months. This pair is concordant because \(0.30>0.20\). B and E are censored, so neither contributes a pair as the earlier event.

There are \(4+3+1=8\) comparable pairs and \(4+2+1=7\) concordant pairs:

\widehat C_{\mathrm{Harrell}}=\frac{7}{8}=0.875.

In this hypothetical sample, 87.5% of the comparable pairs were correctly ranked. This describes the observed pairs in this sample; it is not a statement that 87.5% of individual patients received accurate absolute-risk predictions.

What Censoring Changes

Harrell’s C uses pairs that are comparable in the observed data. But whether a pair is observed as comparable depends on censoring. For example, a patient whose event would have occurred early may be censored before that event, removing pairs that would otherwise have contributed. As a result, Harrell’s C can depend on the censoring distribution and need not estimate the same population quantity across settings with different censoring patterns.

One population target is concordance among pairs whose earlier event occurs by a chosen time \(\tau\). For two independent patients drawn from the target population, define

C_\tau = \frac{ E\!\left[I(T_1<T_2,\ T_1\le\tau)c(r_1,r_2)\right] }{ P(T_1<T_2,\ T_1\le\tau) }.

Here the earlier-event patient is listed first, and true event-time ties are excluded. The time restriction limits the target to event ordering up to \(\tau\). When censoring is independent of the event time and the risk score, inverse-probability-of-censoring weighting can adjust observed comparable pairs toward this target. This method requires that censoring probabilities be estimated reliably and not be zero over the times being evaluated.

Assumptions for the basic censoring adjustment: Patients are independent draws from the target population; censoring is independent of event time and risk score; the censoring survival probabilities used in the weights are correctly estimated; and the relevant probabilities of remaining uncensored are positive. If censoring depends on measured patient characteristics, a marginal censoring model may be inadequate; a conditional censoring model must then be justified and estimated appropriately.

Inverse-Probability Weighting and the Boundary Convention

For an earlier event at time \(t\), observing a comparable pair requires two things: the patient with the event must not be censored before \(t\), and the later patient must remain uncensored strictly beyond \(t\). Define the censoring survival function as \(G(t)=P(C_i>t)\). With the event convention \(\delta_i=I(T_i\le C_i)\), the relevant probabilities are \(G_-(t)=P(C_i\ge t)\) for the event patient and \(G_+(t)=P(C_i>t)\) for the later patient. Under independent censoring across patients, the probability of observing the pair is their product.

P(C_i\ge t,\ C_j>t)=G_-(t)G_+(t).

Thus, a weighted concordance estimator through time \(\tau\) gives each observed pair with earlier event time \(Y_i\) weight \(1/\{\widehat G_-(Y_i)\widehat G_+(Y_i)\}\). It uses the same weight in its numerator and denominator:

\widehat C_{\mathrm{IPCW}}(\tau) = \frac{ \displaystyle\sum_{i\ne j} \frac{I(\delta_i=1,\ Y_i<Y_j,\ Y_i\le\tau)c(r_i,r_j)} {\widehat G_-(Y_i)\widehat G_+(Y_i)} }{ \displaystyle\sum_{i\ne j} \frac{I(\delta_i=1,\ Y_i<Y_j,\ Y_i\le\tau)} {\widehat G_-(Y_i)\widehat G_+(Y_i)} }.

The notation \(\widehat G_-\) and \(\widehat G_+\) denotes estimates of the left- and right-side censoring survival probabilities, respectively. If the censoring-time distribution has no atoms, then \(P(C_i\ge t)=P(C_i>t)\), so the product reduces to \(G(t)^2\). The familiar squared weight \(1/\widehat G(t)^2\) therefore relies on a no-atoms assumption at the relevant times. If censoring can have a point mass at an event time, the two probabilities can differ, and the product form is needed for this strict pair convention.

Identify comparable observed pairs.
For each observed event at \(Y_i\), include only patients \(j\) with \(Y_j>Y_i\), and apply the chosen upper time limit if one is used.

Estimate censoring probabilities.
Estimate \(\widehat G_-(Y_i)\) and \(\widehat G_+(Y_i)\) using a censoring-distribution estimator with the event-time boundary convention stated explicitly.

Weight and compare.
Weight each pair by the inverse product, add its concordance contribution to the numerator, and divide by the sum of pair weights.

The justification for the weighting is direct. For a true ordered pair with \(T_i=t<T_j\), let \(A\) be the event that it is observed as a comparable pair. Under the stated independent-censoring assumptions, \(P(A\mid T_i,T_j,r_i,r_j)=G_-(t)G_+(t)\). Therefore,

E\!\left[ \frac{I(A)}{G_-(t)G_+(t)} \ \middle|\ T_i=t,T_j>t,r_i,r_j \right] = \frac{P(A\mid T_i=t,T_j>t,r_i,r_j)}{G_-(t)G_+(t)} =1.

So the weighted observed-pair contribution has, conditionally, the same expected weight as the underlying true pair. Applying the same weighting to concordance contributions and to all eligible-pair contributions motivates the ratio estimator. If censoring depends on the risk score or event time after accounting for the censoring model, this argument does not hold as written.

Worked Example: A Censoring Atom Changes the Pair Weight

Consider a hypothetical analysis with two event-time groups and comparable pairs observed through \(\tau\). At event time \(t=2\) months, suppose the estimated censoring probabilities are \(\widehat G_-(2)=0.80\) and \(\widehat G_+(2)=0.60\). One observed pair at this time is concordant. At event time \(t=4\) months, suppose both probabilities are 0.60; two pairs are observed, one concordant and one discordant. Assume these are the only pairs.

At month 2, the pair weight is the inverse probability that both required observations are available:

w(2)=\frac{1}{0.80\times0.60} =\frac{1}{0.48} =2.083.

At month 4, the weight for each pair is \(1/(0.60\times0.60)=2.778\). The total weight in the denominator is \(2.083+2(2.778)=7.639\). The weighted concordance numerator is \(2.083+2.778=4.861\), since the month-2 pair and one month-4 pair are concordant:

\widehat C_{\mathrm{IPCW}}(\tau) =\frac{4.861}{7.639} =0.6364.

The calculation uses different left- and right-side probabilities at month 2, as required by the stated pair definition. Replacing the product at month 2 with \(0.80^2\) would give a weight of \(1/0.64=1.563\), which is not the inverse probability of observing this pair when \(\widehat G_+(2)=0.60\). The no-atoms squared-weight formula is not justified for this example.

Interpreting and Checking a Concordance Estimate

A concordance estimate summarizes relative ordering for the pairs included by its definition. It does not evaluate whether a predicted two-year event probability of 0.40 is close to the corresponding observed risk. Any strictly increasing transformation of all risk scores leaves their ordering—and therefore concordance—unchanged, even if the transformed values are not meaningful as probabilities. Absolute-risk calibration is a separate predictive property, not something established by a high C-index.

Check the following before interpreting either statistic:

Score direction: Confirm that larger values mean higher event risk. A reversed score can make a good ranking appear poor.
Pair definition and ties: State how event-time ties and risk-score ties are handled. Different conventions can give different estimates.
Comparable-pair count: A small number of pairs can make the estimate unstable, even if the decimal result looks precise.
Weight size and positivity: IPCW weights become large when estimated censoring survival probabilities are small. Report the evaluation horizon and examine whether there is adequate follow-up support through it.
Censoring model: The IPCW interpretation depends on an appropriate censoring model. If censoring differs across risk-score groups, marginal weights may not correct the selection of observed pairs.
Evaluation data: Evaluate predictions on data not used to fit or select the model, or use a suitable resampling design. Training-set concordance can be optimistic, especially for high-dimensional omics models.

Harrell’s C is simple and directly describes observed comparable pairs, but the set of those pairs can be shaped by censoring. IPCW concordance explicitly targets a censoring-adjusted pair population under additional assumptions and can be unstable when weights are large. Both are ranking measures. A time-dependent ROC analysis asks a time-specific discrimination question, while calibration and Brier-score approaches assess other aspects of predictive accuracy; neither should be inferred from concordance alone.

Common Mistake and Practical Limits

Wrong reasoning: “The event patient and the later patient each need to be uncensored at the event time, so the pair-observation probability is always \(G(t)^2\).” This silently treats \(P(C\ge t)\) and \(P(C>t)\) as equal. They are equal when the censoring distribution has no atom at \(t\), but need not be equal when censoring has a point mass there. Correction: Under the strict pair convention, use \(P(C\ge t)P(C>t)\), estimated by the corresponding left- and right-side probabilities; use a squared probability only with the no-atoms condition.

Concordance also compresses many pairwise comparisons into one number. It can hide poor ranking in a clinically important subgroup or period, and it does not identify which omics features drive performance. Its value can vary with the population’s event-time distribution and with the chosen restriction \(\tau\). For uncertainty assessment, resample patients—not individual pairs—because many pairs share the same patients and are not independent.

Key takeaway: Concordance measures how well a score orders comparable patients by event time. Harrell’s C uses observed comparable pairs; censoring-adjusted concordance weights those pairs under explicit assumptions. Neither statistic alone demonstrates accurate absolute-risk prediction.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

A patient has an observed event at month 3. Another patient is censored at month 5. Are they comparable under the strict convention? Why?
For a comparable pair, the earlier-event patient has risk score 0.4 and the later patient has risk score 0.7. What is the pair’s concordance contribution?
At time \(t\), suppose \(P(C\ge t)=0.9\) and \(P(C>t)=0.5\). What is the pair-observation probability and its inverse weight under independent censoring?
Why can a model have high concordance but poor absolute-risk predictions?
What should be resampled to assess uncertainty in concordance, and why?

Solutions. (1) Yes. The event is observed at month 3 and the second patient’s observed time is later than 3, so the second patient is known to be event-free beyond month 3. (2) The contribution is 0 because the earlier-event patient has the lower risk score, so the pair is discordant. (3) The probability is \(0.9\times0.5=0.45\); the inverse weight is \(1/0.45=2.222\). (4) Concordance depends on ranking and is unchanged by a strictly increasing transformation of scores, so it does not check whether predicted risk magnitudes match observed risks. (5) Resample patients, because pairs reuse patients and are statistically dependent.