Tutorials › Exam PA - Predictive Analytics › Calculating Empirical AUC From Score Pairs

Home  /  Tutorials  /  Exam PA / Classification Metrics and Thresholds

Exam PA · Classification Metrics and Thresholds · Tutorial 720 of 966

Calculating Empirical AUC From Score Pairs

Calculate empirical AUC by comparing every positive score with every negative score, giving tied pairs half credit.

Intermediate 11 min read

What You'll Learn

  • Set up the positive-negative pair count and its denominator
  • Award one point for a correctly ordered pair and half a point for a tie
  • Calculate empirical AUC step by step from small score datasets
  • Check whether scores are oriented so larger values indicate the positive class
  • Recognize common errors in pair counting and tie handling

From Scores to Positive-Negative Comparisons

A classifier can assign a score to each observation without yet assigning a final positive or negative decision. Empirical AUC summarizes how well those scores order the two classes: it compares the score of each positive observation with the score of each negative observation. This makes the calculation possible directly from a small labeled dataset, without selecting a classification threshold.

The direction of the score matters. In this tutorial, a higher score is assumed to indicate stronger evidence for the positive class. If a model uses the opposite convention, first reverse the score direction or define the comparisons accordingly. A pair receives full credit when its positive observation has the higher score, no credit when the negative observation has the higher score, and half credit when the scores tie.

Definition: Suppose a labeled dataset contains \(n_+\) positive observations with scores \(s_1^+,\ldots,s_{n_+}^+\), and \(n_-\) negative observations with scores \(s_1^-,\ldots,s_{n_-}^-\). For each positive-negative pair, assign 1 if \(s_i^+>s_j^-\), 0.5 if \(s_i^+=s_j^-\), and 0 if \(s_i^+<s_j^-\). The empirical AUC is the sum of these pair credits divided by \(n_+n_-\), provided both classes are present.
$$ \widehat{\mathrm{AUC}} = \frac{1}{n_+n_-} \sum_{i=1}^{n_+}\sum_{j=1}^{n_-} \left[ \mathbf{1}(s_i^+>s_j^-) +\frac{1}{2}\mathbf{1}(s_i^+=s_j^-) \right]. $$

Here, \(\mathbf{1}(\text{condition})\) equals 1 when the condition is true and 0 otherwise. The denominator \(n_+n_-\) is the total number of cross-class pairs, not the number of observations and not the number of pairs the classifier orders correctly. The calculation requires observed scores and binary labels; it does not require a parametric distribution for the scores.

A Reliable Pair-Counting Procedure

For a small dataset, the safest approach is to compare each positive score with every negative score. A row-by-row tally makes the denominator visible and helps prevent ties from being treated as either full credit or no credit. The calculation can be organized as follows.

1
Separate the labels.
Write down the positive scores and negative scores, and confirm which class is treated as positive.
2
Check score orientation.
For the formula above, larger scores must indicate stronger evidence for the positive class.
3
Compare all cross-class pairs.
For each positive score, count its wins, ties, and losses against the full list of negative scores.
4
Sum credits and divide.
Add one for each win and one-half for each tie, then divide by \(n_+n_-\).

A loss contributes zero, so it need not appear in the numerator. It should still be included in the total number of pairs. In particular, when multiple observations share the same score, each positive-negative pairing at that score is a separate tie and receives its own half-point.

Worked Example: A Dataset With Several Tied Pairs

A hypothetical review model assigns scores to three positive cases and four negative cases. Larger scores indicate greater evidence for the positive class. The positive scores are 0.90, 0.70, and 0.70; the negative scores are 0.80, 0.70, 0.40, and 0.20. Calculate the empirical AUC. Name the method and identify a counting error to avoid.

Method: positive-negative pair comparison. There are \(n_+=3\) positive cases and \(n_-=4\) negative cases, so the denominator is \(3\times4=12\) pairs.

For the positive score 0.90, comparisons against 0.80, 0.70, 0.40, and 0.20 are all wins. Its credit is \(4\).

For the first positive score 0.70, comparison with 0.80 is a loss, comparison with 0.70 is a tie worth 0.5, and comparisons with 0.40 and 0.20 are wins. Its credit is \(0+0.5+1+1=2.5\).

The second positive score of 0.70 has the same comparisons and also earns \(2.5\). The total credit is \(4+2.5+2.5=9\), so

$$ \widehat{\mathrm{AUC}}=\frac{9}{12}=0.75. $$

Check: There are exactly 12 positive-negative pairs. Of these, nine full-credit-equivalents are accumulated after each tie contributes one-half. The result is within the possible range from 0 to 1. A common mistake would be to count the two ties as full wins, giving \(10/12\); that overstates AUC because a tie does not establish which class is ranked higher.

Why the Pair Formula Works

The pair formula is an empirical version of the ranking interpretation of AUC. Imagine selecting one positive observation uniformly from the observed positives and, independently, one negative observation uniformly from the observed negatives. Every possible positive-negative pair is equally likely: the probability of selecting any particular pair is \(1/(n_+n_-)\). The pair outcome is a win, a tie, or a loss according to the scores.

To find the average outcome over this selection process, add the credit for each possible pair and divide by the number of equally likely pairs. A win contributes 1, a tie contributes 0.5, and a loss contributes 0. The resulting average is exactly the empirical AUC formula. Thus the half-credit rule is not an arbitrary adjustment: it is the conventional treatment that gives a tied positive-negative pair the midpoint between a strict win and a strict loss.

$$ \text{average pair credit} = \frac{ (\text{number of wins}) +\frac12(\text{number of ties}) }{ \text{number of positive-negative pairs} } = \widehat{\mathrm{AUC}}. $$

This justification also clarifies what the statistic summarizes: pairwise ranking among the observed positives and negatives. It does not measure whether scores are calibrated as probabilities, and it does not choose a threshold for converting scores into binary decisions.

Worked Example: Ties and an Easy-to-Miss Denominator

A hypothetical screening model is tested on four labeled records. The positive records have scores 0.82, 0.60, 0.55, and 0.30. The negative records have scores 0.75, 0.60, and 0.40. Larger scores indicate the positive class. Calculate the empirical AUC, showing the contribution from each positive record.

There are four positives and three negatives, so the total number of pairs is \(4\times3=12\). Compare each positive score with all three negative scores:

  • Score 0.82 exceeds 0.75, 0.60, and 0.40: credit \(3\).
  • Score 0.60 loses to 0.75, ties 0.60 for credit \(0.5\), and exceeds 0.40: credit \(0+0.5+1=1.5\).
  • Score 0.55 loses to 0.75 and 0.60, and exceeds 0.40: credit \(1\).
  • Score 0.30 is below all three negative scores: credit \(0\).

The total pair credit is \(3+1.5+1+0=5.5\). Therefore,

$$ \widehat{\mathrm{AUC}}=\frac{5.5}{12}\approx0.4583. $$

The displayed result is rounded to four decimal places. Check: The single tied pair contributes one-half, and all 12 pairs are represented in the denominator. Dividing by four positives alone would be incorrect; each positive is compared with three negatives, not just one.

Worked Example: Reversing a Score That Runs in the Other Direction

A hypothetical triage rule produces a low score for cases more likely to be positive. The positive scores are 0.12, 0.38, and 0.38; the negative scores are 0.05, 0.38, and 0.81. Calculate AUC using the stated score direction. Explain what would go wrong if the higher-is-positive formula were applied without adjustment.

Because lower values indicate the positive class here, a positive-negative pair is correctly ordered when the positive score is lower than the negative score. There are \(3\times3=9\) pairs.

For the positive score 0.12, comparisons with negative scores 0.05, 0.38, and 0.81 give a loss, a win, and a win, respectively: credit \(2\).

For each positive score 0.38, comparison with 0.05 is a loss, comparison with 0.38 is a tie worth 0.5, and comparison with 0.81 is a win. Each earns \(1.5\), so the two such positives earn \(3\) in total.

The total credit is \(2+3=5\), giving

$$ \widehat{\mathrm{AUC}}=\frac{5}{9}\approx0.5556. $$

The result is rounded to four decimal places. Check: Five credit points out of nine pairs is consistent with the three positive records’ contributions. Applying the higher-is-positive rule without adjustment would reverse the ranking comparison and report the complementary value, \(4/9\approx0.4444\), because there are no other outcomes besides wins, losses, and ties. The direction must be settled before counting.

Common Exam Trap: Counting Observations Instead of Pairs

A frequent error is to count how many positive observations appear to be “above” the negatives, then divide by the number of positive observations. That is not the pairwise calculation: one positive score may beat several negatives, and each comparison counts separately.

For example, in the first worked example, the positive score 0.90 beats four negative scores. Counting it as one successful positive observation, then treating each positive as a single trial, would produce a different quantity. The correct count is four pair wins from that one score, because it forms four distinct cross-class pairs. The denominator is 12, not 3.

Other errors to check for before finalizing a calculation include:

  • Omitting tied pairs from the denominator or giving them full credit instead of half credit.
  • Using \(n_++n_-\), the number of observations, instead of \(n_+n_-\), the number of cross-class pairs.
  • Comparing positive-positive or negative-negative scores, which do not enter the pair count.
  • Assuming a larger score means more likely positive when the model’s score has the opposite orientation.
  • Trying to calculate AUC when the dataset contains no positive cases or no negative cases; then \(n_+n_-=0\), so this empirical pair formula is undefined.
Key takeaway: For empirical AUC, compare every positive score with every negative score. Add one for a positive score that is higher, one-half for a tie, and zero for a lower score; divide the total by the number of positive-negative pairs.

Check Your Understanding

For each problem, use the score direction stated and count all positive-negative pairs, including ties.

  1. A model gives positive scores 8 and 5, and negative scores 7 and 4. What is the empirical AUC? (Answer: 0.7500)
  2. Positive scores are 0.90 and 0.60; negative scores are 0.80, 0.60, and 0.20. Higher scores indicate the positive class. What is the empirical AUC? (Answer: 0.7500)
  3. Positive scores are 0.70, 0.50, and 0.20; negative scores are 0.60, 0.50, 0.10, and 0.90. Higher scores indicate the positive class. What is the empirical AUC? (Answer: 0.4583)
  4. A score is lower for cases more likely to be positive. Positive scores are 0.10 and 0.40; negative scores are 0.20, 0.40, and 0.70. Using the stated direction, what is the empirical AUC? (Answer: 0.7500)
  5. Two positive records each have score 3. Three negative records have scores 3, 2, and 1. Higher scores indicate the positive class. What is the empirical AUC? (Answer: 0.8333)

Previous

Next

Back to Tutorials
Clinical Biostats