Discrimination Metrics for Binary Outcomes

Why Discrimination Is a Ranking Question

A polygenic risk score (PRS) assigns each person a numerical score intended to summarize genetic risk for an outcome. When the outcome is binary, a basic evaluation question is whether people who develop the outcome tend to receive higher scores than people who do not. This is a question about discrimination: how well the score separates cases from controls by ranking them.

Discrimination is not the same as accurate risk estimation. A score can rank cases above controls reasonably well while assigning predicted risks that are too high or too low. The area under the receiver operating characteristic curve (ROC AUC) measures ranking ability; threshold-based measures describe classification after choosing a particular cutoff. The two kinds of measures answer different questions.

Definition: Let $Y=1$ indicate a case and $Y=0$ a control, and let $S$ be a score for which higher values are intended to indicate greater risk. The population ROC AUC is $$ \mathrm{AUC}=P(S_1>S_0)+\frac{1}{2}P(S_1=S_0), $$ where $S_1$ and $S_0$ are independent score draws from the case and control populations, respectively. Thus, AUC is the probability that a randomly selected case has a higher score than a randomly selected control, with a tie counted as half a correctly ordered pair.

This definition assumes that case and control labels are defined consistently, that scores are oriented so higher means greater risk, and that the score is evaluated on data not used to select or tune it. The target case and control populations also need to be specified: differences in case severity or control composition can change the score distributions and therefore the AUC. When estimating performance for a target population, the evaluation cases and controls should represent the relevant conditional score distributions in that population. The introductory material on phenotype definition and scale is relevant here: changing the outcome definition can change which individuals count as cases and controls, and hence can change discrimination.

Empirical AUC and Its Ranking Interpretation

For $n_1$ cases and $n_0$ controls in an evaluation sample, the empirical AUC compares every case score with every control score. Define the comparison credit $h(a,b)$ to be 1 if $a>b$, one half if $a=b$, and 0 if $a<b$. Then

\widehat{\mathrm{AUC}} = \frac{1}{n_1n_0} \sum_{i=1}^{n_1}\sum_{j=1}^{n_0} h(S_{1i},S_{0j}).

The denominator is the number of case-control pairs. The numerator is the total credit earned by those pairs under the tie convention. AUC therefore summarizes pairwise ordering, not the average distance between case and control scores. The same ordering, and thus the same AUC, results from any strictly increasing transformation of all scores, such as changing units from one scale to another. Reversing the score direction reverses the ordering; it is not a harmless change of scale.

Worked Example: Calculating AUC With Tied Scores

Suppose a hypothetical evaluation sample has three cases with PRS values $0.9, 0.7, 0.7$ and three controls with values $0.8, 0.7, 0.2$. There are $3\times3=9$ case-control pairs.

The case with score $0.9$ beats all three controls, earning 3 credits. Each case with score $0.7$ beats the control at $0.2$, ties the control at $0.7$, and loses to the control at $0.8$. Each therefore earns $1+\frac{1}{2}+0=1.5$ credits. Total credit is $3+1.5+1.5=6$, so

\widehat{\mathrm{AUC}}=\frac{6}{9}=0.6667.

In this sample, a randomly selected case-control pair receives full correct-ordering credit about two thirds of the time on average, with ties receiving half credit. This is an estimate from the hypothetical sample; it does not mean that two thirds of all cases are correctly classified at a particular cutoff.

The empirical AUC can also be understood as the area under the empirical ROC curve. Each possible threshold gives a point with false-positive rate on the horizontal axis and sensitivity on the vertical axis. Lowering the threshold generally increases both rates. With the half-credit convention, the area of the resulting step curve equals the pairwise formula above: each case-control ordering contributes according to whether the case score is higher, lower, or tied. This equivalence explains why AUC summarizes ranking over all thresholds rather than performance at just one cutoff.

Orient and check the score.
Confirm that larger PRS values are intended to correspond to greater risk, and confirm that the evaluation labels and score are defined consistently.

Compare case-control pairs.
For each pair, award 1 for a case score above the control score, 0 for a lower case score, and one half for a tie.

Average the credits.
Divide total credit by $n_1n_0$. Report the estimate with its evaluation population and, where available, an uncertainty interval.

An AUC of $0.5$ means that the average pairwise ranking is at chance level under this definition; it does not prove that case and control score distributions are identical. An AUC of 1 requires every case score to be strictly greater than every control score. Ties lower the AUC under the half-credit convention. An AUC below $0.5$ can indicate that the score direction is reversed, although the observed value should first be checked for sampling variability and data errors.

Measures at a Chosen Threshold

A threshold turns a continuous score into a binary prediction. For a threshold $c$, classify a person as predicted positive when $S\ge c$, and predicted negative otherwise. In the target population, sensitivity is the probability of a positive prediction among cases, and specificity is the probability of a negative prediction among controls. Their complements and combinations describe the ROC point and related classification summaries.

Definition: At threshold $c$, sensitivity (also called the true-positive rate) is $P(S\ge c\mid Y=1)$. Specificity is $P(S<c\mid Y=0)$, and the false-positive rate is $1-\mathrm{specificity}=P(S\ge c\mid Y=0)$. Balanced accuracy is $(\mathrm{sensitivity}+\mathrm{specificity})/2$, and Youden’s index is $J=\mathrm{sensitivity}+\mathrm{specificity}-1$. These quantities depend on the chosen threshold.

Worked Example: Computing Metrics at One PRS Cutoff

In a hypothetical sample, four cases have scores $0.9, 0.8, 0.6, 0.4$, and four controls have scores $0.7, 0.5, 0.3, 0.2$. Use $c=0.6$, with scores equal to the cutoff classified as positive.

Three of four cases have scores at least $0.6$, so sensitivity is $3/4=0.75$. Three of four controls have scores below $0.6$, so specificity is $3/4=0.75$. The false-positive rate is $1-0.75=0.25$. The ROC point at this cutoff is therefore $(0.25,0.75)$, with false-positive rate first and sensitivity second.

Balanced accuracy is $(0.75+0.75)/2=0.75$, and Youden’s index is $0.75+0.75-1=0.50$. At this cutoff, four of the eight people are classified positive: three cases and one control. The resulting accuracy is $4/8=0.50$ correct? Counting directly, the correctly classified people are the three positive cases and three negative controls, giving $6/8=0.75$. The correct accuracy is therefore $0.75$; the four positive classifications are not the number of correct classifications.

This one ROC point does not determine the full AUC. It describes performance at $c=0.6$ only; other cutoffs give other sensitivity and false-positive-rate pairs.

Balanced accuracy gives equal weight to sensitivity and specificity, regardless of how common cases are in the evaluation population. Ordinary accuracy, by contrast, is the proportion correctly classified and depends on both the threshold and outcome prevalence. Neither balanced accuracy nor Youden’s index chooses a universally appropriate threshold: that choice depends on the intended use and the relative consequences of false positives and false negatives.

Prevalence, Precision, and the Choice of Metric

Precision, also called positive predictive value, is the proportion of predicted positives who are cases. Unlike sensitivity and specificity, precision depends on outcome prevalence as well as on the threshold-specific rates. This distinction matters for PRS evaluation, especially when the outcome is uncommon. A case-control evaluation sample with an artificially high case fraction can make precision appear much higher than it would be in a population with lower prevalence.

Worked Example: Why Precision Changes With Prevalence

Consider a hypothetical population of 1,000 people in which 1% are cases: 10 cases and 990 controls. At a specified PRS threshold, suppose sensitivity is $0.80$ and specificity is $0.90$. Applying those rates gives $0.80\times10=8$ true positives and $0.10\times990=99$ false positives.

Precision is the proportion of predicted positives who are cases:

\mathrm{Precision} = \frac{\mathrm{true\ positives}} {\mathrm{true\ positives}+\mathrm{false\ positives}} = \frac{8}{8+99} = \frac{8}{107} \approx 0.07477.

Thus, about 7.5% of people classified positive at this threshold are cases in this hypothetical population. The threshold still has 80% sensitivity and 90% specificity, but those values do not imply high precision when the outcome is rare. If the same conditional score distributions are sampled with a different case fraction, sensitivity, specificity, and AUC can remain the same while precision changes.

The precision-recall curve plots precision against recall, where recall is sensitivity, across thresholds. It can be useful when positive outcomes are rare because it directly displays the fraction of predicted positives that are cases. Its baseline and values depend on prevalence, so the case fraction and evaluation design must be reported. The term “area under the precision-recall curve” can be calculated using different conventions; a report should state the convention used. ROC AUC and precision-recall summaries are complementary rather than interchangeable: the former emphasizes case-control ranking, while the latter makes the positive predictive yield visible at the evaluated prevalence.

What a Change in AUC Does and Does Not Establish

A higher AUC for one PRS than another indicates better average case-control ranking in the population represented by the evaluation data, subject to sampling uncertainty and the study design. It does not establish that the score produces accurate individual probabilities, that a particular clinical cutoff is useful, or that use of the score improves outcomes. It also does not establish that genetic variation captured by the score causes the outcome.

AUC is prevalence-independent in the limited sense that its population definition uses the conditional score distributions among cases and controls, not the proportion of cases. A case-control sampling design can therefore estimate AUC if the sampled cases and controls adequately represent those conditional distributions. But changes in case mix, control selection, phenotype definition, or score measurement can change those distributions and the resulting AUC. The metric is not automatically transportable to a different population merely because the outcome prevalence is similar or different.

A difference between two estimated AUCs is also subject to uncertainty. If the same people receive both scores, the estimates are paired; comparing them should account for that dependence. Confidence intervals or an appropriate paired comparison are needed before treating a numerical difference as evidence of a reproducible performance gain. Hyperparameter choices must be made without using the final test data; otherwise, the reported discrimination can be optimistic. The earlier material on nested validation addresses this separation of tuning and evaluation.

Measure	What it summarizes	Key qualification
ROC AUC	Average case-control ordering across score thresholds	Does not identify a useful cutoff or establish accurate predicted risks
Sensitivity and specificity	Case detection and control rejection at one threshold	Both depend on the threshold
Balanced accuracy and Youden’s index	Combined sensitivity and specificity at one threshold	Do not account for outcome prevalence or application-specific consequences
Precision and the precision-recall curve	Case yield among positive predictions, across thresholds for the curve	Depend on prevalence; report the evaluation case fraction and method

Diagnostics and Common Pitfalls

Before interpreting a discrimination result, check that the outcome labels and score direction are correct, that each person contributes as intended, and that the evaluation data were not used to tune the score. Examine whether the estimate is supported by enough cases and controls to be informative; a point estimate without uncertainty can conceal substantial imprecision. If the intended use depends on a particular threshold, report its sensitivity, specificity, and other relevant consequences rather than relying on AUC alone.

Reversing the score. AUC below $0.5$ is not automatically evidence of a genuinely harmful ranking. Verify that higher scores were intended to indicate higher risk and that the case label was not reversed.
Calling AUC a classification rate. An AUC of $0.70$ does not mean 70% of people are correctly classified. It is a pairwise ranking summary, with ties receiving half credit.
Ignoring ties. Under the half-credit convention, a tie is neither a full win nor a full loss. In particular, AUC equals 1 only when every case score is strictly above every control score.
Treating a threshold metric as universal. Sensitivity and specificity describe one cutoff. Moving the cutoff changes them, and choosing a cutoff from the test set can make its apparent performance optimistic.
Reading precision from AUC. AUC does not determine how many positive predictions are true cases. Precision depends on prevalence as well as on threshold-specific sensitivity and specificity.
Overinterpreting an AUC increase. A small estimated increase alone does not establish a statistically reliable or practically important improvement. Consider uncertainty, validation design, and the intended prediction task.

Key takeaway: ROC AUC describes average case-control ranking; threshold-specific measures describe performance at a chosen cutoff. Neither, on its own, establishes calibration, clinical benefit, or causation. Interpret each metric in the population and outcome definition for which it was evaluated.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A case score is tied with a control score. How much credit does that pair contribute to empirical AUC, and what does this imply about a sample in which every case ties every control?
At a threshold, there are 18 true positives, 6 false negatives, 72 true negatives, and 24 false positives. Calculate sensitivity, specificity, balanced accuracy, and Youden’s index.
Why can two populations have the same sensitivity and specificity at a threshold but different precision?
A PRS has AUC $0.5$ in an evaluation sample. What does this say about average pairwise ranking, and what does it not prove?

Solutions. (1) The tied pair contributes one half. If every case ties every control, every pair contributes one half, so empirical AUC is $0.5$, not 1. (2) Sensitivity is $18/(18+6)=18/24=0.75$. Specificity is $72/(72+24)=72/96=0.75$. Balanced accuracy is $(0.75+0.75)/2=0.75$, and Youden’s index is $0.75+0.75-1=0.50$. (3) Precision depends on the numbers of cases and controls as well as the threshold-specific rates. With a lower case prevalence, false positives can outnumber true positives by more, reducing precision even when sensitivity and specificity are unchanged. (4) The estimate indicates chance-level average pairwise ranking in that sample. It does not prove identical case and control score distributions, nor does it rule out sampling uncertainty or issues with score direction, phenotype definition, or evaluation design.