What AUC Says About a Pair of Cases
A ROC curve describes how sensitivity and false-positive rate change as the classification threshold changes. Its area under the curve, or AUC, can also be read as a ranking probability: it measures how often the model gives a randomly selected positive case a more positive score than a randomly selected negative case. This interpretation focuses on score ordering rather than on a particular threshold.
To state the interpretation precisely, imagine drawing one case from the positive class and, independently, one case from the negative class. Let \(S_+\) and \(S_-\) denote their scores. The score must be oriented so that a larger value indicates stronger evidence for the positive class. The two draws are independent, but they need not have the same score distribution.
The probabilities refer to the score distributions for the two classes in the population or validation sample being considered. AUC is not the probability that a particular case is positive given its score. That latter quantity is a conditional probability and depends, among other things, on the prevalence of the positive class.
Why Ties Receive Half Credit
If scores are continuous, exact ties typically have probability zero, so the formula reduces to \(P(S_+>S_-)\). Ties matter when scores are discrete, rounded, or assigned to categories. Giving each tie half credit treats the two tied cases as equally likely to be ranked first if a tie were broken at random.
The half-credit convention also agrees with the ROC area. A threshold sweep adds cases with a tied score as a block: some positive cases enter the predicted-positive group at the same score as some negative cases. If the ROC path connects the points before and after that block by a straight segment, the segment gives the block its midpoint contribution. Thus, the block is treated as though its positive and negative members were ordered at random relative to one another. Each positive-negative tie contributes one-half to the overall ranking probability.
A Short Derivation of the Ranking Interpretation
First suppose the scores have continuous distributions, so ties have probability zero. As the threshold \(c\) moves downward, a case is predicted positive when its score is at least \(c\). At that threshold, the true-positive rate is \(P(S_+\ge c)\), and the false-positive rate is \(P(S_-\ge c)\). The area under the ROC curve can be written as the accumulated true-positive rate as the false-positive rate increases.
For a continuous negative-score distribution with density \(f_-\), an infinitesimal increase in false-positive rate around score \(c\) is \(f_-(c)\,dc\) as the threshold moves through that score. Therefore,
The middle equality follows by averaging the probability \(P(S_+\ge c)\) over a negative score \(c=S_-\). With continuous scores, \(P(S_+=S_-)=0\), so \(P(S_+\ge S_-)=P(S_+>S_-)\). When scores have atoms, threshold sweeps can add tied positive and negative cases together; assigning half credit to their pairwise ties matches the ROC segment convention. The resulting general formula is the definition above.
Worked Example: A Score Where Smaller Values Indicate Positives
A hypothetical screening model reports an ordinal score of 0, 1, or 2, with smaller scores indicating stronger evidence of a positive case. For a randomly selected positive case, the probabilities of scores 0, 1, and 2 are 0.20, 0.50, and 0.30. For a randomly selected negative case, those probabilities are 0.50, 0.30, and 0.20. Find the AUC in the model’s stated direction, including ties.
Method. Since smaller values indicate positives, count pairs with \(S_+<S_-\) as strict wins, then add half the probability of a tie.
A positive score of 0 is lower than negative scores 1 and 2. The probability of those pairs is \(0.20(0.30+0.20)=0.10\). A positive score of 1 is lower than a negative score of 2, contributing \(0.50(0.20)=0.10\). Thus,
The tie probability is the sum of the three same-score probabilities:
Therefore, the AUC in the stated, lower-is-more-positive direction is \(0.20+\frac12(0.31)=0.355\). As a check, the strict loss probability is \(0.49\): positive score 1 against negative score 0 contributes \(0.50(0.50)=0.25\), and positive score 2 against negative scores 0 or 1 contributes \(0.30(0.50+0.30)=0.24\). The three pair categories sum to \(0.20+0.31+0.49=1\).
Common mistake avoided: Do not count only positive score 1 against negative score 2 as a strict win. Positive score 0 is also lower than negative scores 1 and 2, and its contribution is \(0.10\). If the score is instead negated so that larger values indicate positives, the AUC becomes \(0.49+\frac12(0.31)=0.645\), the complementary ranking direction.
Computing AUC From Score Distributions
When the class-conditional score distributions are stated in a table or described by a probability model, the ranking interpretation gives a direct route to AUC. For discrete scores, consider each possible positive score and the negative scores it outranks, then add the probabilities of all those pairings. Separately add the probabilities of equal-score pairings. The pair probabilities use products because the positive and negative cases are drawn independently.
Worked Example: Continuous Scores With No Ties
Suppose a hypothetical model produces independent scores \(S_+\sim N(2,1)\) for positive cases and \(S_-\sim N(0,1)\) for negative cases. The second parameter is the variance, so each score has variance 1. Larger scores indicate positives. Find the AUC, rounded to four decimal places.
Method. With continuous normal scores, ties have probability zero. Define the score difference \(D=S_+-S_-\). The difference of independent normal variables is normal, with mean equal to the difference of means and variance equal to the sum of variances:
The model ranks the positive case above the negative case exactly when \(D>0\). Standardizing gives
Thus, among independently selected positive-negative pairs, about 92.14% are ordered correctly by this score model. As a check, the difference has positive mean and the resulting AUC is above 0.5, as expected for a score whose positive-class distribution is shifted upward. The common mistake is to use the variance of the difference as 1; independence means the variances add, giving 2.
Worked Example: A Score That Represents Only a Group
A hypothetical model assigns score 0 to cases in group A and score 1 to cases in group B. Larger scores indicate positives. Among positive cases, 30% are in group A and 70% are in group B. Among negative cases, 70% are in group A and 30% are in group B. Find the AUC, including the ties.
Method. A positive-negative pair is a strict win only when the positive case is in group B and the negative case is in group A. By independent selection, that probability is \(0.70(0.70)=0.49\).
A tie occurs when both cases are in A or both are in B:
The AUC is \(0.49+\frac12(0.42)=0.70\). As a check, strict losses occur when the positive case is in A and the negative case is in B, with probability \(0.30(0.30)=0.09\); wins, ties, and losses total \(0.49+0.42+0.09=1\).
Common mistake avoided: Reporting 0.49 as the AUC ignores ties. The model does not order the cases within a group, so tied pairs contribute half credit rather than no credit. The different group proportions by class determine the pairwise probabilities; the score values 0 and 1 alone do not determine AUC.
How to Interpret and Use the Number
AUC summarizes the model’s ranking across all possible thresholds, rather than its classification decisions at one chosen threshold. An AUC of 0.70 means that a randomly selected positive-negative pair receives a correct ordering 70% of the time when ties are counted as half. It does not mean that 70% of all cases are classified correctly, nor that a case with a particular score has a 70% probability of being positive.
AUC is unaffected by a strictly increasing transformation of all scores because such a transformation preserves their ordering and their ties. For example, replacing a score \(S\) by \(10+3S\) preserves every strict comparison and equality, so it preserves AUC. A strictly decreasing transformation reverses the ranking direction. With the same tie convention, reversing the direction changes AUC to \(1-\operatorname{AUC}\): strict wins become strict losses, strict losses become wins, and ties remain ties.
The measure also has limitations. It does not show whether predicted probabilities are calibrated, whether the model performs well at a threshold that matters operationally, or whether the model is useful when error costs differ. AUC averages ranking performance over thresholds and can conceal behavior in a region of the ROC curve that matters most to a decision-maker. It is a ranking summary, not a complete model assessment.
- Check direction first. Confirm whether higher or lower scores indicate the positive class before interpreting an AUC.
- Keep ties in the calculation. Each tied positive-negative pair contributes one-half, not zero or one.
- State the pair being sampled. AUC compares one positive draw with one negative draw; it is not an overall case-level accuracy rate.
- Do not infer calibration. AUC uses score ordering, not whether a predicted probability matches an observed event frequency.
Check Your Understanding
For each problem, use independent positive and negative draws, check the score direction, and include half credit for ties where relevant.
- A continuous model has independent scores \(S_+\sim N(1,1)\) and \(S_-\sim N(0,1)\), with larger values indicating positives. What is the AUC, rounded to four decimals? (Answer: \(\Phi(1/\sqrt{2})\approx 0.7602\))
- A two-level score is higher for positives. Positive cases have probabilities 0.60 for score 1 and 0.40 for score 0; negative cases have probabilities 0.20 for score 1 and 0.80 for score 0. Find the strict-win probability, tie probability, and AUC. (Answer: strict wins \(0.48\), ties \(0.44\), AUC \(0.70\))
- A model has discrete scores 0, 1, and 2, with higher scores indicating positives. Positive score probabilities are \(0.10, 0.40, 0.50\); negative score probabilities are \(0.50, 0.30, 0.20\), in the same order. Find its AUC. (Answer: strict wins \(0.60\), ties \(0.27\), AUC \(0.735\))
- A scoring system has AUC \(0.82\) when larger scores indicate positives. If every score is multiplied by \(-1\) and the direction is then interpreted as larger indicating positives, what is the new AUC? (Answer: \(1-0.82=0.18\))
- A model assigns score 0 to group A and score 1 to group B. Positive cases are 40% in A and 60% in B; negative cases are 60% in A and 40% in B. Higher scores indicate positives. Find the AUC and verify the tie contribution. (Answer: strict wins \(0.36\), ties \(0.48\), AUC \(0.60\))